{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Structural-Based extractive summarization\n",
    "\n",
    "以往的抽取式摘要算法：\n",
    "1. 没有有考虑到**文章结构**对于关键词的影响 (只是使用启发式算法)\n",
    "2. 使用外部语料作为重要度的来源 (如CNN等语料训练tf-idf)\n",
    "3. 没有考虑到词之间的多种语义关系（如wordNet, word embedding）\n",
    "\n",
    "\n",
    "本文内容\n",
    "1. 本文首先验证了，在domain-specific的摘要系统中，无法获取准确的词重要度，依据文章内部的资源进行重要度抽取（与两个假定的冲突），并通过t-test检验这种差距的显著性\n",
    "2. 使用多种方法基于文章结构测定词的重要性\n",
    "3. 基于多种语义方式构建多种keywords之间的关系，并使用rank公式来找出关键词\n",
    "4. 利用sentence shortening技术，无需background corpus和POS等词性标注就能很好地完成文本摘要工作\n",
    "5. 这种重要性公式对于抽取文章维度、提供文章语义也有帮助。\n",
    "\n",
    "实验对比\n",
    "1. 与LexRank, TextRank, Bert-centroid based 对比\n",
    "2. 使用TokenToMe进行分词抽取\n",
    "3. 使用词形还原后的\n",
    "4. 去除停用词\n",
    "5. 与RST系统对比\n",
    "6. 与WordNet系统对比\n",
    "7. 如果只在introduction和conclusion中进行摘要和全文摘要的对比\n",
    "\n",
    "结果展示\n",
    "1. 展示一个抽取出来的摘要\n",
    "2. 展示抽取出来的关键词\n",
    "\n",
    "图表\n",
    "1. 证明关键词比例的\n",
    "2. 显示四种方式在各种ROUGE评分下的PR曲线\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 导入依赖\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "from code.data_loader import Corpus, tokenize, segment\n",
    "import matplotlib.pyplot as plt\n",
    "from metrics import rouge_score\n",
    "from pathlib import Path\n",
    "from functools import partial\n",
    "\n",
    "tokenizer_ = partial(tokenize, remove_stop=True)\n",
    "corpus = Corpus(tokenizer_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "item = corpus.items[0]\n",
    "all_sentences = segment(item.introduction + item.sections + item.conclusion)\n",
    "fe_sentences = segment(item.introduction + item.conclusion)\n",
    "\n",
    "truth = tokenizer_(item.abstract)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 四种方法的PR曲线\n",
    "\n",
    "1. tf\n",
    "2. tf-idf\n",
    "3. chi2\n",
    "4. mutual information\n",
    "5. log likelihood"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Invalid RGBA argument: 'dark'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m~/.conda/envs/ccf/lib/python3.6/site-packages/matplotlib/colors.py\u001b[0m in \u001b[0;36mto_rgba\u001b[0;34m(c, alpha)\u001b[0m\n\u001b[1;32m    165\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 166\u001b[0;31m         \u001b[0mrgba\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_colors_full_map\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcache\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0malpha\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    167\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mKeyError\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# Not in cache, or unhashable.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: ('dark', None)",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m~/.conda/envs/ccf/lib/python3.6/site-packages/IPython/core/formatters.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, obj)\u001b[0m\n\u001b[1;32m    339\u001b[0m                 \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    340\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 341\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mprinter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    342\u001b[0m             \u001b[0;31m# Finally look for special method names\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    343\u001b[0m             \u001b[0mmethod\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_real_method\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprint_method\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/ccf/lib/python3.6/site-packages/IPython/core/pylabtools.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(fig)\u001b[0m\n\u001b[1;32m    239\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    240\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;34m'png'\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mformats\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 241\u001b[0;31m         \u001b[0mpng_formatter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfor_type\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mFigure\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mfig\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mprint_figure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'png'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    242\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;34m'retina'\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mformats\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m'png2x'\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mformats\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    243\u001b[0m         \u001b[0mpng_formatter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfor_type\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mFigure\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mfig\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mretina_figure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/ccf/lib/python3.6/site-packages/IPython/core/pylabtools.py\u001b[0m in \u001b[0;36mprint_figure\u001b[0;34m(fig, fmt, bbox_inches, **kwargs)\u001b[0m\n\u001b[1;32m    123\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    124\u001b[0m     \u001b[0mbytes_io\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mBytesIO\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 125\u001b[0;31m     \u001b[0mfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcanvas\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprint_figure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbytes_io\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    126\u001b[0m     \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbytes_io\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetvalue\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    127\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfmt\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'svg'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/ccf/lib/python3.6/site-packages/matplotlib/backend_bases.py\u001b[0m in \u001b[0;36mprint_figure\u001b[0;34m(self, filename, dpi, facecolor, edgecolor, orientation, format, **kwargs)\u001b[0m\n\u001b[1;32m   2210\u001b[0m                     \u001b[0morientation\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0morientation\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2211\u001b[0m                     \u001b[0mdryrun\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2212\u001b[0;31m                     **kwargs)\n\u001b[0m\u001b[1;32m   2213\u001b[0m                 \u001b[0mrenderer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfigure\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_cachedRenderer\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2214\u001b[0m                 \u001b[0mbbox_inches\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfigure\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_tightbbox\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrenderer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/ccf/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py\u001b[0m in \u001b[0;36mprint_png\u001b[0;34m(self, filename_or_obj, *args, **kwargs)\u001b[0m\n\u001b[1;32m    515\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    516\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mprint_png\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfilename_or_obj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 517\u001b[0;31m         \u001b[0mFigureCanvasAgg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    518\u001b[0m         \u001b[0mrenderer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_renderer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    519\u001b[0m         \u001b[0moriginal_dpi\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrenderer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdpi\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/ccf/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py\u001b[0m in \u001b[0;36mdraw\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    435\u001b[0m             \u001b[0;31m# if toolbar:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    436\u001b[0m             \u001b[0;31m#     toolbar.set_cursor(cursors.WAIT)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 437\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfigure\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrenderer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    438\u001b[0m             \u001b[0;31m# A GUI class may be need to update a window using this draw, so\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    439\u001b[0m             \u001b[0;31m# don't forget to call the superclass.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/ccf/lib/python3.6/site-packages/matplotlib/artist.py\u001b[0m in \u001b[0;36mdraw_wrapper\u001b[0;34m(artist, renderer, *args, **kwargs)\u001b[0m\n\u001b[1;32m     53\u001b[0m                 \u001b[0mrenderer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart_filter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     54\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 55\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0martist\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrenderer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     56\u001b[0m         \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     57\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0martist\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_agg_filter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/ccf/lib/python3.6/site-packages/matplotlib/figure.py\u001b[0m in \u001b[0;36mdraw\u001b[0;34m(self, renderer)\u001b[0m\n\u001b[1;32m   1491\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1492\u001b[0m             mimage._draw_list_compositing_images(\n\u001b[0;32m-> 1493\u001b[0;31m                 renderer, self, artists, self.suppressComposite)\n\u001b[0m\u001b[1;32m   1494\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1495\u001b[0m             \u001b[0mrenderer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose_group\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'figure'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/ccf/lib/python3.6/site-packages/matplotlib/image.py\u001b[0m in \u001b[0;36m_draw_list_compositing_images\u001b[0;34m(renderer, parent, artists, suppress_composite)\u001b[0m\n\u001b[1;32m    139\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mnot_composite\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mhas_images\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    140\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0ma\u001b[0m \u001b[0;32min\u001b[0m \u001b[0martists\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 141\u001b[0;31m             \u001b[0ma\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrenderer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    142\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    143\u001b[0m         \u001b[0;31m# Composite any adjacent images together\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/ccf/lib/python3.6/site-packages/matplotlib/artist.py\u001b[0m in \u001b[0;36mdraw_wrapper\u001b[0;34m(artist, renderer, *args, **kwargs)\u001b[0m\n\u001b[1;32m     53\u001b[0m                 \u001b[0mrenderer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart_filter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     54\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 55\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0martist\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrenderer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     56\u001b[0m         \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     57\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0martist\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_agg_filter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/ccf/lib/python3.6/site-packages/matplotlib/axes/_base.py\u001b[0m in \u001b[0;36mdraw\u001b[0;34m(self, renderer, inframe)\u001b[0m\n\u001b[1;32m   2633\u001b[0m             \u001b[0mrenderer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstop_rasterizing\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2634\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2635\u001b[0;31m         \u001b[0mmimage\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_draw_list_compositing_images\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrenderer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0martists\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2636\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2637\u001b[0m         \u001b[0mrenderer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose_group\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'axes'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/ccf/lib/python3.6/site-packages/matplotlib/image.py\u001b[0m in \u001b[0;36m_draw_list_compositing_images\u001b[0;34m(renderer, parent, artists, suppress_composite)\u001b[0m\n\u001b[1;32m    139\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mnot_composite\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mhas_images\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    140\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0ma\u001b[0m \u001b[0;32min\u001b[0m \u001b[0martists\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 141\u001b[0;31m             \u001b[0ma\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrenderer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    142\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    143\u001b[0m         \u001b[0;31m# Composite any adjacent images together\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/ccf/lib/python3.6/site-packages/matplotlib/artist.py\u001b[0m in \u001b[0;36mdraw_wrapper\u001b[0;34m(artist, renderer, *args, **kwargs)\u001b[0m\n\u001b[1;32m     53\u001b[0m                 \u001b[0mrenderer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart_filter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     54\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 55\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0martist\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrenderer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     56\u001b[0m         \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     57\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0martist\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_agg_filter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/ccf/lib/python3.6/site-packages/matplotlib/lines.py\u001b[0m in \u001b[0;36mdraw\u001b[0;34m(self, renderer)\u001b[0m\n\u001b[1;32m    760\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_set_gc_clip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    761\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 762\u001b[0;31m                 \u001b[0mlc_rgba\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmcolors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_rgba\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_color\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_alpha\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    763\u001b[0m                 \u001b[0mgc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_foreground\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlc_rgba\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0misRGBA\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    764\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/ccf/lib/python3.6/site-packages/matplotlib/colors.py\u001b[0m in \u001b[0;36mto_rgba\u001b[0;34m(c, alpha)\u001b[0m\n\u001b[1;32m    166\u001b[0m         \u001b[0mrgba\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_colors_full_map\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcache\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0malpha\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    167\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mKeyError\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# Not in cache, or unhashable.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 168\u001b[0;31m         \u001b[0mrgba\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_to_rgba_no_colorcycle\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0malpha\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    169\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    170\u001b[0m             \u001b[0m_colors_full_map\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcache\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0malpha\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrgba\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/ccf/lib/python3.6/site-packages/matplotlib/colors.py\u001b[0m in \u001b[0;36m_to_rgba_no_colorcycle\u001b[0;34m(c, alpha)\u001b[0m\n\u001b[1;32m    210\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    211\u001b[0m             \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 212\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Invalid RGBA argument: {!r}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0morig_c\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    213\u001b[0m     \u001b[0;31m# tuple color.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    214\u001b[0m     \u001b[0mc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Invalid RGBA argument: 'dark'"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 1440x720 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def prec_recall(sent_toks, metric=\"rouge-1\", desired_length=60):\n",
    "    choices = []\n",
    "    for toks in sent_toks:\n",
    "        choices += toks\n",
    "        if len(choices) >= desired_length:\n",
    "            break\n",
    "    summary = \" \".join(choices)\n",
    "    score = rouge_score(truth, summary)[0][metric]\n",
    "    \n",
    "    return score[\"p\"], score[\"r\"], score[\"f\"]\n",
    "\n",
    "def prec_recall_list(sent_toks, metric=\"rouge-1\", mean=False):\n",
    "    ps, rs = [], []\n",
    "    for desired_length in range(5, 300, 10):\n",
    "        p, r, f = prec_recall(sent_toks, metric, desired_length)\n",
    "        ps.append(p)\n",
    "        rs.append(r)\n",
    "        \n",
    "    return ps, rs\n",
    "\n",
    "prec_recall_list(item.summarize(\"chi2\"))\n",
    "\n",
    "fig = plt.figure(figsize=(20, 10))\n",
    "\n",
    "x_lim = {\n",
    "    \"rouge-1\": [0, 0.7],\n",
    "    \"rouge-2\": [0, 0.2],\n",
    "    \"rouge-l\": [0, 0.6],\n",
    "}\n",
    "\n",
    "y_lim = {\n",
    "    \"rouge-1\": [0, 0.4],\n",
    "    \"rouge-2\": [0, 0.2],\n",
    "    \"rouge-l\": [0, 0.3],\n",
    "}\n",
    "\n",
    "for i, metric in enumerate([\"rouge-1\", \"rouge-2\", \"rouge-l\"], 1):\n",
    "    axis = fig.add_subplot(2, 2, i)\n",
    "    axis.set_title(metric)\n",
    "    for label, color in zip([\"tf\", \"idf\", \"chi2\", \"gi\", \"llr\"], \n",
    "                           [\"green\", \"darkorange\", \"brown\", \"blue\", \"dark\"]):\n",
    "        ps, rs = prec_recall_list(item.summarize(label), metric)\n",
    "        axis.plot(rs, ps, label=label, color=color)\n",
    "        \n",
    "    p, r = lex_pr(metric)\n",
    "    axis.plot(r, p, label=\"lex\", color=\"red\")\n",
    "    axis.legend()\n",
    "    axis.set_xlim(x_lim[metric])\n",
    "    axis.set_ylim(y_lim[metric])\n",
    "    \n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LexRank"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lexrank import STOPWORDS, LexRank\n",
    "documents = []\n",
    "documents_dir = Path('../bbc/tech')\n",
    "\n",
    "for file_path in documents_dir.glob('*.txt'):\n",
    "    with file_path.open(mode='rt', encoding='utf-8') as fp:\n",
    "        documents.append(fp.readlines())\n",
    "lxr = LexRank(documents, stopwords=STOPWORDS['en'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'rouge-1': {'f': 0.13432835379817348, 'p': 0.1, 'r': 0.20454545454545456},\n",
       "  'rouge-2': {'f': 0.023809519832767103,\n",
       "   'p': 0.01639344262295082,\n",
       "   'r': 0.043478260869565216},\n",
       "  'rouge-l': {'f': 0.11093806805283159, 'p': 0.1, 'r': 0.20454545454545456}}]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lex_all_summary = lxr.get_summary(all_sentences, summary_size=5, threshold=.1)\n",
    "rouge_score(truth, \" \".join(lex_all_summary))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'rouge-1': {'f': 0.16374268781642218,\n",
       "   'p': 0.09395973154362416,\n",
       "   'r': 0.6363636363636364},\n",
       "  'rouge-2': {'f': 0.021089629445300805,\n",
       "   'p': 0.011472275334608031,\n",
       "   'r': 0.13043478260869565},\n",
       "  'rouge-l': {'f': 0.08544650043361231,\n",
       "   'p': 0.08389261744966443,\n",
       "   'r': 0.5681818181818182}}]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lex_fe_summary = lxr.get_summary(fe_sentences, summary_size=20, threshold=.1)\n",
    "rouge_score(truth, \" \".join(lex_fe_summary))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lex_pr(metric):\n",
    "    p, r = prec_recall_list([tokenizer_(summ, join=False) for summ in lex_fe_summary], metric)\n",
    "    return p, r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def llr_f(inputs):\n",
    "    X = vectorizer.fit_transform(inputs.data).toarray()\n",
    "    Y = np.array(inputs.target)\n",
    "    Y = LabelBinarizer().fit_transform(Y)\n",
    "    if Y.shape[1] == 1:\n",
    "        Y = np.append(1 - Y, Y, axis=1)\n",
    "\n",
    "    col = X / np.sum(a, axis=0);\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 416,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = vectorizer.fit_transform(inputs.data).toarray()\n",
    "Y = np.array(inputs.target)\n",
    "Y = LabelBinarizer().fit_transform(Y)\n",
    "if Y.shape[1] == 1:\n",
    "    Y = np.append(1 - Y, Y, axis=1)\n",
    "freq = np.dot(Y.T, X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 417,
   "metadata": {},
   "outputs": [],
   "source": [
    "col_prob = freq / np.sum(freq, axis=0)\n",
    "row_prob = freq / np.sum(freq, axis=1).reshape(-1, 1)\n",
    "N = freq.sum()\n",
    "row_sum = np.sum(freq, axis=1).reshape(-1, 1)\n",
    "\n",
    "def log(prob):\n",
    "    prob[prob == 0] = prob[prob == 0] + 1 / N\n",
    "    return np.log(prob)\n",
    "\n",
    "\n",
    "row_entropy = np.sum(freq * log(row_prob) + (row_sum - freq) * log(1 - row_prob), axis=0)\n",
    "col_entropy = np.sum(freq * log(col_prob), axis=0) + np.sum((row_sum - freq) * log((row_sum - freq) / np.sum(row_sum - freq, axis=0)), axis=0)\n",
    "mat_entropy = np.sum(freq * log(freq / N), axis=0) + np.sum((row_sum - freq) * log((row_sum - freq) / N), axis=0)\n",
    "\n",
    "llrv = - 2 * (mat_entropy - row_entropy - col_entropy)\n",
    "return llrv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 420,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 384,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 6],\n",
       "       [24]])"
      ]
     },
     "execution_count": 384,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "row_sum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 382,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0. , 0.5, 1. , ..., 1. , 1. , 1. ],\n",
       "       [1. , 0.5, 0. , ..., 0. , 0. , 0. ]])"
      ]
     },
     "execution_count": 382,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 383,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.00108342, 0.00216685, 0.00325027],\n",
       "       [0.05501618, 0.00970874, 0.01294498]])"
      ]
     },
     "execution_count": 383,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.dot(Y.T, X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 326,
   "metadata": {},
   "outputs": [],
   "source": [
    "def logLikelihoodRatio(k11, k12, k21, k22):\n",
    "        rowEntropy = entropy(k11, k12) + entropy(k21, k22);\n",
    "        columnEntropy = entropy(k11, k21) + entropy(k12, k22);\n",
    "        matrixEntropy = entropy(k11, k12, k21, k22);\n",
    "        return 2 * (matrixEntropy - rowEntropy - columnEntropy);\n",
    "\n",
    "def entropy(*elements):\n",
    "        s = sum(elements)\n",
    "        result = 0.0;\n",
    "    \n",
    "        for x in elements:\n",
    "            # int zeroFlag = (x == 0 ? 1 : 0);\n",
    "            result += x * log((x + 0) / s);\n",
    "        \n",
    "        return -result;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 394,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.07062661671220027"
      ]
     },
     "execution_count": 394,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logLikelihoodRatio(1, 5, 2, 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 356,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([\n",
    "    [1, 2, 3],\n",
    "    [17, 3, 4]\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 357,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.05555556, 0.4       , 0.42857143],\n",
       "       [0.94444444, 0.6       , 0.57142857]])"
      ]
     },
     "execution_count": 357,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col = a / np.sum(a, axis=0);\n",
    "col"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 358,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.16666667, 0.33333333, 0.5       ],\n",
       "       [0.70833333, 0.125     , 0.16666667]])"
      ]
     },
     "execution_count": 358,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "row = a / np.sum(a, axis=1).reshape(-1, 1)\n",
    "row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 359,
   "metadata": {},
   "outputs": [],
   "source": [
    "N = a.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 360,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-17.19066129, -12.86156888, -14.9723521 ])"
      ]
     },
     "execution_count": 360,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re = np.sum(a * log(row) + (row_sum - a) * log(1 - row), axis=0) ;\n",
    "re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 336,
   "metadata": {},
   "outputs": [],
   "source": [
    "col_sum = np.sum(a, axis=0).reshape(1, -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 337,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[17,  3,  4],\n",
       "       [ 1,  2,  3]])"
      ]
     },
     "execution_count": 337,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col_sum - a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 338,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1,  2,  3],\n",
       "       [17,  3,  4]])"
      ]
     },
     "execution_count": 338,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 339,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 5,  4,  3],\n",
       "       [-8,  6,  5]])"
      ]
     },
     "execution_count": 339,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "row_sum - a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 340,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.16666667, 0.33333333, 0.5       ],\n",
       "       [0.70833333, 0.125     , 0.16666667]])"
      ]
     },
     "execution_count": 340,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 341,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/lee/.conda/envs/ccf/lib/python3.6/site-packages/ipykernel_launcher.py:1: RuntimeWarning: invalid value encountered in log\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([         nan, -10.09517501, -10.07286264])"
      ]
     },
     "execution_count": 341,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ce = np.sum(a * log(col), axis=0) + np.sum((row_sum - a) * log((row_sum - a) / np.sum(row_sum - a, axis=0)), axis=0);\n",
    "ce"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 342,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/lee/.conda/envs/ccf/lib/python3.6/site-packages/ipykernel_launcher.py:1: RuntimeWarning: invalid value encountered in log\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([         nan, -30.04009524, -30.83391999])"
      ]
     },
     "execution_count": 342,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "me = np.sum(a * log(a / N), axis=0) + np.sum((row_sum - a) * log((row_sum - a) / N), axis=0);\n",
    "me"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 343,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([         nan, -18.17264448, -17.04705721])"
      ]
     },
     "execution_count": 343,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "2 * (me - re - ce)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 344,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-17.565898049855566"
      ]
     },
     "execution_count": 344,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1 * log(1 / 15) + 2 * log(2 / 15) + 5 * log(5 / 15) + 7 * log(7 / 15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 345,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-10.059861696782747"
      ]
     },
     "execution_count": 345,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log(1 / 3) + 2 * log(2 / 3) + 5 * log(5 / 12) + 7 * log(7 / 12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 346,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.0986122886681098"
      ]
     },
     "execution_count": 346,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log(1 / 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 347,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.16666667, 0.33333333, 0.5       ],\n",
       "       [0.70833333, 0.125     , 0.16666667]])"
      ]
     },
     "execution_count": 347,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 348,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.79175947, -2.19722458, -2.07944154],\n",
       "       [-5.86228827, -6.23832463, -7.16703788]])"
      ]
     },
     "execution_count": 348,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a * log(row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 349,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-2.703367253197828"
      ]
     },
     "execution_count": 349,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log(1 / 6) + 5 * log(5 / 6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 350,
   "metadata": {},
   "outputs": [],
   "source": [
    "row_sum = np.sum(a, axis=1).reshape(-1, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 351,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-7.470723044716719"
      ]
     },
     "execution_count": 351,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log(1 / 6) + 5 * log(5 / 6) + 2 * log(2 / 9) + 7 * log(7 / 9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 352,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-10.059861696782747"
      ]
     },
     "execution_count": 352,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log(1 / 3) + 2 * log(2 / 3) + 5 * log(5 / 12) + 7 * log(7 / 12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 353,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": 353,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "N"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 354,
   "metadata": {},
   "outputs": [],
   "source": [
    "from code.info import llr_f, chi2_f"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 355,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'1349837': 3.857142857142857,\n",
       " '1502': 1.0582010582010584,\n",
       " '1750': 0.2592592592592593,\n",
       " '1800': 0.2592592592592593,\n",
       " '1836': 1.0582010582010584,\n",
       " '1849': 0.5185185185185186,\n",
       " '2006': 0.2592592592592593,\n",
       " '383': 0.2592592592592593,\n",
       " '480': 0.2592592592592593,\n",
       " '528': 0.2592592592592593,\n",
       " 'accuracy': 0.2980599647266314,\n",
       " 'accuracybaseline': 0.2592592592592593,\n",
       " 'active': 3.857142857142857,\n",
       " 'adaptation': 4.407407407407407,\n",
       " 'add': 0.5185185185185186,\n",
       " 'adding': 0.2592592592592593,\n",
       " 'again': 0.2592592592592593,\n",
       " 'al': 0.5417989417989418,\n",
       " 'algorithm': 1.0582010582010584,\n",
       " 'algorithms': 3.857142857142857,\n",
       " 'all': 0.2592592592592593,\n",
       " 'alternative': 0.5961199294532628,\n",
       " 'alternatives': 0.2592592592592593,\n",
       " 'analysis': 3.895943562610229,\n",
       " 'analytic': 0.2592592592592593,\n",
       " 'analytically': 2.1164021164021167,\n",
       " 'annotations': 1.0582010582010584,\n",
       " 'appear': 0.2592592592592593,\n",
       " 'applicability': 3.857142857142857,\n",
       " 'application': 0.2592592592592593,\n",
       " 'applied': 0.7777777777777778,\n",
       " 'apply': 1.1523809523809525,\n",
       " 'applying': 0.2592592592592593,\n",
       " 'approach': 0.2980599647266314,\n",
       " 'approaches': 2.1164021164021167,\n",
       " 'arbitrary': 0.2592592592592593,\n",
       " 'areandwhere': 0.2592592592592593,\n",
       " 'as': 3.857142857142857,\n",
       " 'asin': 0.2592592592592593,\n",
       " 'assume': 0.2592592592592593,\n",
       " 'assuming': 0.2592592592592593,\n",
       " 'autoencoder': 2.123204837490552,\n",
       " 'autoencoders': 2.123204837490552,\n",
       " 'award': 3.857142857142857,\n",
       " 'aware': 3.857142857142857,\n",
       " 'bag': 1.0582010582010584,\n",
       " 'baldridge': 0.2592592592592593,\n",
       " 'base': 0.5185185185185186,\n",
       " 'based': 0.2592592592592593,\n",
       " 'baseline': 0.2592592592592593,\n",
       " 'belonging': 1.0370370370370372,\n",
       " 'best': 0.5185185185185186,\n",
       " 'better': 3.857142857142857,\n",
       " 'biases': 0.2592592592592593,\n",
       " 'bilinear': 0.2592592592592593,\n",
       " 'binary': 3.857142857142857,\n",
       " 'blitzer': 0.7777777777777778,\n",
       " 'brahe': 0.5185185185185186,\n",
       " 'build': 3.857142857142857,\n",
       " 'case': 0.2592592592592593,\n",
       " 'cases': 0.2592592592592593,\n",
       " 'centering': 0.2592592592592593,\n",
       " 'challenging': 0.2592592592592593,\n",
       " 'chance': 0.2592592592592593,\n",
       " 'characterize': 7.714285714285714,\n",
       " 'chen': 1.1523809523809525,\n",
       " 'choose': 0.2592592592592593,\n",
       " 'chooses': 3.857142857142857,\n",
       " 'choosing': 0.2592592592592593,\n",
       " 'chosen': 0.5185185185185186,\n",
       " 'classification': 3.857142857142857,\n",
       " 'closed': 0.2592592592592593,\n",
       " 'co': 0.2592592592592593,\n",
       " 'common': 3.857142857142857,\n",
       " 'community': 3.857142857142857,\n",
       " 'compare': 0.7777777777777778,\n",
       " 'comparison': 3.857142857142857,\n",
       " 'composed': 0.2592592592592593,\n",
       " 'computation': 0.2592592592592593,\n",
       " 'compute': 1.0370370370370372,\n",
       " 'computed': 0.2592592592592593,\n",
       " 'computing': 0.5185185185185186,\n",
       " 'computingwhere': 0.2592592592592593,\n",
       " 'concerns': 0.2592592592592593,\n",
       " 'conditional': 0.2592592592592593,\n",
       " 'conducted': 0.2592592592592593,\n",
       " 'consequently': 3.857142857142857,\n",
       " 'consider': 0.0010582010582010574,\n",
       " 'considerably': 0.2980599647266314,\n",
       " 'considered': 0.2592592592592593,\n",
       " 'consuming': 0.2592592592592593,\n",
       " 'context': 0.5185185185185186,\n",
       " 'contribution': 1.0582010582010584,\n",
       " 'corpus': 1.2962962962962963,\n",
       " 'correspondence': 0.2980599647266314,\n",
       " 'corrupt': 0.2592592592592593,\n",
       " 'corrupted': 1.5555555555555556,\n",
       " 'corrupting': 0.2592592592592593,\n",
       " 'corruption': 0.2592592592592593,\n",
       " 'cost': 0.5185185185185186,\n",
       " 'count': 0.5185185185185186,\n",
       " 'create': 0.2592592592592593,\n",
       " 'creating': 0.2592592592592593,\n",
       " 'crf': 0.5185185185185186,\n",
       " 'crfsuite': 0.2592592592592593,\n",
       " 'current': 7.714285714285714,\n",
       " 'data': 3.111111111111111,\n",
       " 'dataset': 0.047619047619047616,\n",
       " 'datasets': 3.895943562610229,\n",
       " 'decided': 0.2592592592592593,\n",
       " 'deep': 3.857142857142857,\n",
       " 'define': 1.0582010582010584,\n",
       " 'defined': 0.2592592592592593,\n",
       " 'degrades': 0.2592592592592593,\n",
       " 'demonstrate': 3.857142857142857,\n",
       " 'denoised': 0.2592592592592593,\n",
       " 'denoiser': 0.2592592592592593,\n",
       " 'denoising': 2.304761904761905,\n",
       " 'denote': 0.2592592592592593,\n",
       " 'denotes': 0.2592592592592593,\n",
       " 'dense': 0.2592592592592593,\n",
       " 'dependent': 3.857142857142857,\n",
       " 'depends': 0.2592592592592593,\n",
       " 'describe': 0.5185185185185186,\n",
       " 'designed': 0.2980599647266314,\n",
       " 'designing': 3.857142857142857,\n",
       " 'development': 0.5185185185185186,\n",
       " 'dhillon': 0.2592592592592593,\n",
       " 'diagonal': 1.5555555555555556,\n",
       " 'dimension': 0.2592592592592593,\n",
       " 'dimensional': 1.0370370370370372,\n",
       " 'dimensionality': 0.5185185185185186,\n",
       " 'directly': 3.857142857142857,\n",
       " 'distribution': 0.5185185185185186,\n",
       " 'distributions': 0.5185185185185186,\n",
       " 'divide': 0.5185185185185186,\n",
       " 'divided': 0.5185185185185186,\n",
       " 'documents': 1.0582010582010584,\n",
       " 'domain': 3.6872427983539096,\n",
       " 'domains': 0.49441504997060554,\n",
       " 'dominated': 0.2592592592592593,\n",
       " 'dos': 0.2592592592592593,\n",
       " 'dramatic': 3.857142857142857,\n",
       " 'draw': 0.2592592592592593,\n",
       " 'drawn': 0.2592592592592593,\n",
       " 'drop': 0.2592592592592593,\n",
       " 'dropout': 0.8185941043083902,\n",
       " 'easier': 3.857142857142857,\n",
       " 'efficiency': 3.857142857142857,\n",
       " 'efficient': 3.857142857142857,\n",
       " 'efficiently': 0.2592592592592593,\n",
       " 'elements': 0.7777777777777778,\n",
       " 'eliminate': 0.2592592592592593,\n",
       " 'eliminates': 3.857142857142857,\n",
       " 'emphasize': 3.857142857142857,\n",
       " 'english': 0.2592592592592593,\n",
       " 'entire': 0.5185185185185186,\n",
       " 'entries': 0.5185185185185186,\n",
       " 'entry': 0.5185185185185186,\n",
       " 'epochs': 0.2592592592592593,\n",
       " 'equal': 0.5185185185185186,\n",
       " 'equations': 0.047619047619047616,\n",
       " 'equivalent': 0.2592592592592593,\n",
       " 'estimate': 0.2592592592592593,\n",
       " 'estimated': 0.2592592592592593,\n",
       " 'estimation': 0.2592592592592593,\n",
       " 'evaluate': 0.2592592592592593,\n",
       " 'evaluating': 3.857142857142857,\n",
       " 'evaluation': 0.2592592592592593,\n",
       " 'evolving': 3.857142857142857,\n",
       " 'exactly': 0.5185185185185186,\n",
       " 'example': 3.857142857142857,\n",
       " 'expectation': 0.2592592592592593,\n",
       " 'expectations': 0.5185185185185186,\n",
       " 'experiments': 1.2962962962962963,\n",
       " 'explain': 0.2592592592592593,\n",
       " 'explicitly': 3.857142857142857,\n",
       " 'exploit': 3.895943562610229,\n",
       " 'extend': 0.2592592592592593,\n",
       " 'factor': 0.2592592592592593,\n",
       " 'fail': 3.857142857142857,\n",
       " 'faster': 3.895943562610229,\n",
       " 'feature': 5.023088023088023,\n",
       " 'features': 0.4821428571428572,\n",
       " 'feedback': 3.857142857142857,\n",
       " 'field': 0.2592592592592593,\n",
       " 'fifty': 0.2592592592592593,\n",
       " 'figure': 0.2592592592592593,\n",
       " 'finally': 0.2592592592592593,\n",
       " 'find': 1.0582010582010584,\n",
       " 'fine': 3.857142857142857,\n",
       " 'fire': 0.2592592592592593,\n",
       " 'firing': 0.2592592592592593,\n",
       " 'focused': 0.5185185185185186,\n",
       " 'focuses': 0.2592592592592593,\n",
       " 'for': 0.2592592592592593,\n",
       " 'form': 0.5185185185185186,\n",
       " 'foundation': 3.857142857142857,\n",
       " 'framework': 0.2592592592592593,\n",
       " 'free': 0.5185185185185186,\n",
       " 'full': 0.2592592592592593,\n",
       " 'functions': 3.857142857142857,\n",
       " 'fundamental': 3.857142857142857,\n",
       " 'fα': 0.2592592592592593,\n",
       " 'fβ': 0.2592592592592593,\n",
       " 'general': 0.5185185185185186,\n",
       " 'generally': 0.2592592592592593,\n",
       " 'generate': 0.2592592592592593,\n",
       " 'grained': 3.857142857142857,\n",
       " 'graphs': 0.2592592592592593,\n",
       " 'grow': 3.857142857142857,\n",
       " 'guo': 0.2592592592592593,\n",
       " 'high': 0.5185185185185186,\n",
       " 'historical': 0.014697236919459145,\n",
       " 'hmm': 0.2592592592592593,\n",
       " 'hold': 0.5185185185185186,\n",
       " 'hope': 3.857142857142857,\n",
       " 'huang': 0.2592592592592593,\n",
       " 'hyper': 0.7777777777777778,\n",
       " 'idea': 3.895943562610229,\n",
       " 'ideas': 3.857142857142857,\n",
       " 'identical': 0.2592592592592593,\n",
       " 'implementation': 0.2980599647266314,\n",
       " 'improvement': 0.2592592592592593,\n",
       " 'improvements': 3.857142857142857,\n",
       " 'improving': 0.2592592592592593,\n",
       " 'in': 0.0010582010582010574,\n",
       " 'include': 0.7777777777777778,\n",
       " 'including': 0.7777777777777778,\n",
       " 'incorporate': 0.2592592592592593,\n",
       " 'incorporating': 0.2592592592592593,\n",
       " 'increasing': 1.0582010582010584,\n",
       " 'indicates': 0.2592592592592593,\n",
       " 'individual': 1.0582010582010584,\n",
       " 'induced': 0.2592592592592593,\n",
       " 'input': 0.7777777777777778,\n",
       " 'inputs': 0.2592592592592593,\n",
       " 'insert': 0.2592592592592593,\n",
       " 'instance': 0.2980599647266314,\n",
       " 'instances': 0.047619047619047616,\n",
       " 'interest': 1.0582010582010584,\n",
       " 'intuition': 0.2592592592592593,\n",
       " 'intuitive': 3.857142857142857,\n",
       " 'inversion': 0.2592592592592593,\n",
       " 'investigate': 3.857142857142857,\n",
       " 'involve': 3.857142857142857,\n",
       " 'involves': 0.2592592592592593,\n",
       " 'is': 0.2592592592592593,\n",
       " 'it': 0.2592592592592593,\n",
       " 'italian': 0.2592592592592593,\n",
       " 'known': 1.0582010582010584,\n",
       " 'labeled': 0.2980599647266314,\n",
       " 'labeling': 0.2980599647266314,\n",
       " 'language': 3.895943562610229,\n",
       " 'languages': 0.2592592592592593,\n",
       " 'large': 3.895943562610229,\n",
       " 'latent': 0.5185185185185186,\n",
       " 'layer': 0.2592592592592593,\n",
       " 'leads': 0.5185185185185186,\n",
       " 'learn': 0.2592592592592593,\n",
       " 'learned': 3.857142857142857,\n",
       " 'learning': 0.8941798941798943,\n",
       " 'level': 0.2592592592592593,\n",
       " 'leverage': 0.2592592592592593,\n",
       " 'linear': 0.2592592592592593,\n",
       " 'log': 0.2592592592592593,\n",
       " 'longer': 0.2592592592592593,\n",
       " 'loss': 0.2592592592592593,\n",
       " 'lossif': 0.2592592592592593,\n",
       " 'low': 0.7777777777777778,\n",
       " 'machine': 0.2592592592592593,\n",
       " 'magnitude': 1.0582010582010584,\n",
       " 'manually': 0.2592592592592593,\n",
       " 'marginalization': 1.0582010582010584,\n",
       " 'marginalize': 0.2980599647266314,\n",
       " 'marginalized': 3.174603174603174,\n",
       " 'matrices': 0.5185185185185186,\n",
       " 'matrix': 3.3703703703703707,\n",
       " 'mda': 0.09523809523809523,\n",
       " 'mean': 0.2592592592592593,\n",
       " 'media': 3.857142857142857,\n",
       " 'method': 0.5185185185185186,\n",
       " 'methods': 0.0010582010582010574,\n",
       " 'middle': 0.5185185185185186,\n",
       " 'minimizing': 0.2592592592592593,\n",
       " 'model': 0.5185185185185186,\n",
       " 'modern': 0.2592592592592593,\n",
       " 'moon': 0.2592592592592593,\n",
       " 'motivated': 0.2592592592592593,\n",
       " 'multiple': 0.2592592592592593,\n",
       " 'multiplication': 0.2592592592592593,\n",
       " 'naive': 0.5185185185185186,\n",
       " 'national': 3.857142857142857,\n",
       " 'natural': 7.714285714285714,\n",
       " 'ne': 0.7777777777777778,\n",
       " 'networks': 0.2592592592592593,\n",
       " 'neural': 0.2592592592592593,\n",
       " 'newstext': 0.2592592592592593,\n",
       " 'next': 0.2592592592592593,\n",
       " 'nlp': 2.1164021164021167,\n",
       " 'nogueira': 0.2592592592592593,\n",
       " 'noise': 0.00529100529100528,\n",
       " 'noising': 7.791887125220458,\n",
       " 'nonlinearity': 0.2592592592592593,\n",
       " 'normalization': 0.2592592592592593,\n",
       " 'normalized': 0.2592592592592593,\n",
       " 'note': 0.2592592592592593,\n",
       " 'novel': 0.2592592592592593,\n",
       " 'number': 2.1164021164021167,\n",
       " 'observed': 3.857142857142857,\n",
       " 'obtaining': 1.0582010582010584,\n",
       " 'occurence': 0.2592592592592593,\n",
       " 'of': 5.721088435374149,\n",
       " 'off': 0.7777777777777778,\n",
       " 'older': 1.0582010582010584,\n",
       " 'on': 0.2592592592592593,\n",
       " 'optimization': 0.2592592592592593,\n",
       " 'order': 3.857142857142857,\n",
       " 'orders': 0.2592592592592593,\n",
       " 'ordinary': 0.2592592592592593,\n",
       " 'original': 3.174603174603174,\n",
       " 'our': 0.2592592592592593,\n",
       " 'outer': 0.5185185185185186,\n",
       " 'outperforms': 0.2592592592592593,\n",
       " 'output': 0.2592592592592593,\n",
       " 'paper': 3.895943562610229,\n",
       " 'parameter': 0.2592592592592593,\n",
       " 'parameters': 1.2962962962962963,\n",
       " 'parsing': 3.857142857142857,\n",
       " 'part': 2.1164021164021167,\n",
       " 'passing': 0.2592592592592593,\n",
       " 'pca': 0.7777777777777778,\n",
       " 'pennacchiotti': 0.2592592592592593,\n",
       " 'perform': 1.0582010582010584,\n",
       " 'performance': 0.2592592592592593,\n",
       " 'periods': 0.2592592592592593,\n",
       " 'pilot': 0.2592592592592593,\n",
       " 'pivot': 1.2962962962962963,\n",
       " 'pivots': 0.5185185185185186,\n",
       " 'portuguese': 0.2980599647266314,\n",
       " 'positive': 0.2592592592592593,\n",
       " 'posterior': 0.2592592592592593,\n",
       " 'predicting': 0.2592592592592593,\n",
       " 'prediction': 0.2592592592592593,\n",
       " 'presence': 0.2592592592592593,\n",
       " 'presented': 0.2592592592592593,\n",
       " 'presents': 0.5185185185185186,\n",
       " 'preserved': 0.2592592592592593,\n",
       " 'preserving': 0.2592592592592593,\n",
       " 'previous': 0.2980599647266314,\n",
       " 'prior': 0.2592592592592593,\n",
       " 'probability': 2.0740740740740744,\n",
       " 'problem': 0.17006802721088438,\n",
       " 'process': 7.714285714285714,\n",
       " 'processes': 0.2592592592592593,\n",
       " 'processing': 7.714285714285714,\n",
       " 'produced': 3.857142857142857,\n",
       " 'product': 0.5185185185185186,\n",
       " 'progressively': 0.2592592592592593,\n",
       " 'project': 0.2592592592592593,\n",
       " 'projecting': 0.2592592592592593,\n",
       " 'projection': 0.7777777777777778,\n",
       " 'projections': 0.5185185185185186,\n",
       " 'propose': 1.0582010582010584,\n",
       " 'proposed': 0.2980599647266314,\n",
       " 'provide': 3.857142857142857,\n",
       " 'qα': 0.7777777777777778,\n",
       " 'qβ': 0.7777777777777778,\n",
       " 'random': 0.5185185185185186,\n",
       " 'randomly': 7.2433862433862455,\n",
       " 'rapidly': 3.857142857142857,\n",
       " 'rare': 0.2592592592592593,\n",
       " 'ratio': 0.2592592592592593,\n",
       " 'ratnaparkhi': 0.2592592592592593,\n",
       " 'real': 0.2592592592592593,\n",
       " 'realistic': 3.857142857142857,\n",
       " 'reasoning': 0.2592592592592593,\n",
       " 'recall': 0.2592592592592593,\n",
       " 'reconstruct': 0.5185185185185186,\n",
       " 'reconstructing': 0.2592592592592593,\n",
       " 'reconstruction': 0.5185185185185186,\n",
       " 'reconstructions': 0.2592592592592593,\n",
       " 'reduce': 0.2592592592592593,\n",
       " 'refer': 0.2592592592592593,\n",
       " 'regularization': 0.2592592592592593,\n",
       " 'relevant': 3.857142857142857,\n",
       " 'relies': 0.2592592592592593,\n",
       " 'rely': 3.857142857142857,\n",
       " 'remote': 0.2592592592592593,\n",
       " 'remove': 3.857142857142857,\n",
       " 'representation': 0.5961199294532628,\n",
       " 'representations': 0.5185185185185186,\n",
       " 'requires': 1.0582010582010584,\n",
       " 'rescale': 0.2592592592592593,\n",
       " 'rescaling': 0.2592592592592593,\n",
       " 'resistant': 3.857142857142857,\n",
       " 'results': 0.2592592592592593,\n",
       " 'review': 0.2592592592592593,\n",
       " 'reviewers': 3.857142857142857,\n",
       " 'robust': 3.857142857142857,\n",
       " 'robustness': 0.2592592592592593,\n",
       " 'roughly': 0.2592592592592593,\n",
       " 'row': 0.2592592592592593,\n",
       " 'sacrificing': 3.857142857142857,\n",
       " 'santos': 0.2592592592592593,\n",
       " 'scaled': 0.5185185185185186,\n",
       " 'scatter': 0.7777777777777778,\n",
       " 'scenario': 0.2592592592592593,\n",
       " 'scheme': 0.2592592592592593,\n",
       " 'science': 3.857142857142857,\n",
       " 'scl': 0.05643738977072312,\n",
       " 'scramble': 0.2592592592592593,\n",
       " 'scrambling': 0.047619047619047616,\n",
       " 'secondary': 3.857142857142857,\n",
       " 'select': 0.2592592592592593,\n",
       " 'selecting': 0.2592592592592593,\n",
       " 'selects': 3.857142857142857,\n",
       " 'sentiment': 3.857142857142857,\n",
       " 'sequence': 0.2980599647266314,\n",
       " 'set': 1.5555555555555556,\n",
       " 'sets': 0.2592592592592593,\n",
       " 'setting': 0.2592592592592593,\n",
       " 'settings': 0.2592592592592593,\n",
       " 'sgd': 0.2592592592592593,\n",
       " 'showing': 3.857142857142857,\n",
       " 'simple': 0.5185185185185186,\n",
       " 'simpler': 3.857142857142857,\n",
       " 'simplicity': 3.857142857142857,\n",
       " 'single': 1.0582010582010584,\n",
       " 'slowed': 0.2592592592592593,\n",
       " 'slower': 0.2592592592592593,\n",
       " 'smaller': 0.2592592592592593,\n",
       " 'social': 3.857142857142857,\n",
       " 'solution': 2.1164021164021167,\n",
       " 'solutions': 1.0370370370370372,\n",
       " 'solve': 0.5185185185185186,\n",
       " 'solving': 0.2980599647266314,\n",
       " 'source': 0.7777777777777778,\n",
       " 'space': 1.0582010582010584,\n",
       " 'spaces': 7.2433862433862455,\n",
       " 'specific': 0.7777777777777778,\n",
       " 'spectral': 0.2592592592592593,\n",
       " 'speech': 2.1164021164021167,\n",
       " 'speedups': 3.857142857142857,\n",
       " 'spirit': 3.857142857142857,\n",
       " 'split': 0.2592592592592593,\n",
       " 'square': 0.2592592592592593,\n",
       " 'squared': 0.2592592592592593,\n",
       " 'stacking': 0.2592592592592593,\n",
       " 'stale': 3.857142857142857,\n",
       " 'standard': 0.05643738977072312,\n",
       " 'statistics': 0.2592592592592593,\n",
       " 'step': 3.857142857142857,\n",
       " 'structural': 0.2980599647266314,\n",
       " 'structure': 7.2433862433862455,\n",
       " 'structured': 7.407407407407406,\n",
       " 'styles': 3.857142857142857,\n",
       " 'sub': 0.2592592592592593,\n",
       " 'subset': 0.5185185185185186,\n",
       " 'subsets': 0.5185185185185186,\n",
       " 'substantial': 3.857142857142857,\n",
       " 'suffix': 3.857142857142857,\n",
       " 'sum': 0.5185185185185186,\n",
       " 'supervised': 0.7777777777777778,\n",
       " 'supported': 3.857142857142857,\n",
       " 'supports': 0.2592592592592593,\n",
       " 'survives': 0.2592592592592593,\n",
       " 'syntactic': 2.1164021164021167,\n",
       " 'synthetic': 3.857142857142857,\n",
       " 'system': 0.2980599647266314,\n",
       " 'systems': 1.0582010582010584,\n",
       " 's𝐖s': 0.2592592592592593,\n",
       " 'table': 0.7777777777777778,\n",
       " 'tackle': 0.2592592592592593,\n",
       " 'tagged': 0.2592592592592593,\n",
       " 'tagger': 0.7777777777777778,\n",
       " 'tagging': 1.1523809523809525,\n",
       " 'tags': 0.2592592592592593,\n",
       " 'tanh': 0.5185185185185186,\n",
       " 'target': 1.2962962962962963,\n",
       " 'task': 0.2592592592592593,\n",
       " 'tasks': 4.750264550264551,\n",
       " 'technique': 0.2592592592592593,\n",
       " 'techniques': 1.0582010582010584,\n",
       " 'template': 3.132863021751912,\n",
       " 'templates': 0.05643738977072312,\n",
       " 'temporally': 0.2592592592592593,\n",
       " 'tend': 3.857142857142857,\n",
       " 'terms': 0.2592592592592593,\n",
       " 'test': 0.7777777777777778,\n",
       " 'testing': 0.5185185185185186,\n",
       " 'text': 0.5185185185185186,\n",
       " 'texts': 7.791887125220458,\n",
       " 'the': 0.2592592592592593,\n",
       " 'third': 0.2592592592592593,\n",
       " 'thousands': 3.857142857142857,\n",
       " 'three': 0.5185185185185186,\n",
       " 'time': 0.2592592592592593,\n",
       " 'times': 0.5185185185185186,\n",
       " 'token': 0.7777777777777778,\n",
       " 'topic': 0.2592592592592593,\n",
       " 'total': 0.7777777777777778,\n",
       " 'toutanova': 3.857142857142857,\n",
       " 'train': 0.2592592592592593,\n",
       " 'trained': 3.857142857142857,\n",
       " 'training': 0.32010582010582017,\n",
       " 'transfer': 0.2592592592592593,\n",
       " 'transform': 7.714285714285714,\n",
       " 'transformed': 0.2592592592592593,\n",
       " 'transition': 0.2592592592592593,\n",
       " 'treated': 0.2592592592592593,\n",
       " 'trend': 0.2592592592592593,\n",
       " 'tunable': 0.2592592592592593,\n",
       " 'tune': 0.5185185185185186,\n",
       " 'tycho': 0.5185185185185186,\n",
       " 'type': 0.5185185185185186,\n",
       " 'types': 0.047619047619047616,\n",
       " 'unchanged': 0.5185185185185186,\n",
       " 'uncorrupted': 0.2592592592592593,\n",
       " 'uniform': 0.2592592592592593,\n",
       " 'unlike': 0.2592592592592593,\n",
       " 'unsupervised': 3.895943562610229,\n",
       " 'valued': 0.2592592592592593,\n",
       " 'values': 0.5185185185185186,\n",
       " 'variance': 0.2592592592592593,\n",
       " 'vector': 0.2592592592592593,\n",
       " 'version': 1.2962962962962963,\n",
       " 'versions': 0.5185185185185186,\n",
       " 'view': 0.2592592592592593,\n",
       " 'we': 0.047619047619047616,\n",
       " 'weight': 0.2592592592592593,\n",
       " 'well': 2.1164021164021167,\n",
       " 'will': 0.49441504997060554,\n",
       " 'word': 2.123204837490552,\n",
       " 'words': 1.0582010582010584,\n",
       " 'work': 0.27286470143613,\n",
       " 'write': 0.5185185185185186,\n",
       " 'writing': 3.857142857142857,\n",
       " 'written': 0.2592592592592593,\n",
       " 'xiao': 0.2592592592592593,\n",
       " 'yates': 0.2592592592592593,\n",
       " 'year': 0.2592592592592593,\n",
       " 'years': 3.857142857142857,\n",
       " 'yt': 0.5185185185185186,\n",
       " 'zanzotto': 0.2592592592592593,\n",
       " 'ℝd': 0.7777777777777778,\n",
       " '𝐐α': 0.2592592592592593,\n",
       " '𝐖s': 0.2592592592592593,\n",
       " '𝐖𝐗': 0.2592592592592593,\n",
       " '𝐗s': 0.2592592592592593,\n",
       " '𝐗𝐗': 0.2592592592592593,\n",
       " '𝐱i': 3.111111111111111,\n",
       " '𝐱n': 0.5185185185185186}"
      ]
     },
     "execution_count": 355,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dict(chi2_f(inputs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'1349837': 13.81750955863044,\n",
       " '1502': -0.0,\n",
       " '1750': -13.81750955863044,\n",
       " '1800': -13.81750955863044,\n",
       " '1836': -0.0,\n",
       " '1849': -13.81750955863044,\n",
       " '2006': -13.81750955863044,\n",
       " '383': -13.81750955863044,\n",
       " '480': -13.81750955863044,\n",
       " '528': -13.81750955863044,\n",
       " 'accuracy': -1.3833010954077654,\n",
       " 'accuracybaseline': -13.81750955863044,\n",
       " 'active': 13.81750955863044,\n",
       " 'adaptation': -0.7118957467189511,\n",
       " 'add': -13.81750955863044,\n",
       " 'adding': -13.81750955863044,\n",
       " 'again': -13.81750955863044,\n",
       " 'al': -1.6907952445714285,\n",
       " 'algorithm': -0.0,\n",
       " 'algorithms': 13.81750955863044,\n",
       " 'all': -13.81750955863044,\n",
       " 'alternative': -1.3833010954077654,\n",
       " 'alternatives': -13.81750955863044,\n",
       " 'analysis': 1.3833010954077654,\n",
       " 'analytic': -13.81750955863044,\n",
       " 'analytically': -0.0,\n",
       " 'annotations': -0.0,\n",
       " 'appear': -13.81750955863044,\n",
       " 'applicability': 13.81750955863044,\n",
       " 'application': -13.81750955863044,\n",
       " 'applied': -13.81750955863044,\n",
       " 'apply': -0.8092670144572764,\n",
       " 'applying': -13.81750955863044,\n",
       " 'approach': -1.3833010954077654,\n",
       " 'approaches': -0.0,\n",
       " 'arbitrary': -13.81750955863044,\n",
       " 'areandwhere': -13.81750955863044,\n",
       " 'as': 13.81750955863044,\n",
       " 'asin': -13.81750955863044,\n",
       " 'assume': -13.81750955863044,\n",
       " 'assuming': -13.81750955863044,\n",
       " 'autoencoder': -0.5741998552952308,\n",
       " 'autoencoders': -0.5741998552952308,\n",
       " 'award': 13.81750955863044,\n",
       " 'aware': 13.81750955863044,\n",
       " 'bag': -0.0,\n",
       " 'baldridge': -13.81750955863044,\n",
       " 'base': -13.81750955863044,\n",
       " 'based': -13.81750955863044,\n",
       " 'baseline': -13.81750955863044,\n",
       " 'belonging': -13.81750955863044,\n",
       " 'best': -13.81750955863044,\n",
       " 'better': 13.81750955863044,\n",
       " 'biases': -13.81750955863044,\n",
       " 'bilinear': -13.81750955863044,\n",
       " 'binary': 13.81750955863044,\n",
       " 'blitzer': -13.81750955863044,\n",
       " 'brahe': -13.81750955863044,\n",
       " 'build': 13.81750955863044,\n",
       " 'case': -13.81750955863044,\n",
       " 'cases': -13.81750955863044,\n",
       " 'centering': -13.81750955863044,\n",
       " 'challenging': -13.81750955863044,\n",
       " 'chance': -13.81750955863044,\n",
       " 'characterize': 13.81750955863044,\n",
       " 'chen': -0.8092670144572764,\n",
       " 'choose': -13.81750955863044,\n",
       " 'chooses': 13.81750955863044,\n",
       " 'choosing': -13.81750955863044,\n",
       " 'chosen': -13.81750955863044,\n",
       " 'classification': 13.81750955863044,\n",
       " 'closed': -13.81750955863044,\n",
       " 'co': -13.81750955863044,\n",
       " 'common': 13.81750955863044,\n",
       " 'community': 13.81750955863044,\n",
       " 'compare': -13.81750955863044,\n",
       " 'comparison': 13.81750955863044,\n",
       " 'composed': -13.81750955863044,\n",
       " 'computation': -13.81750955863044,\n",
       " 'compute': -13.81750955863044,\n",
       " 'computed': -13.81750955863044,\n",
       " 'computing': -13.81750955863044,\n",
       " 'computingwhere': -13.81750955863044,\n",
       " 'concerns': -13.81750955863044,\n",
       " 'conditional': -13.81750955863044,\n",
       " 'conducted': -13.81750955863044,\n",
       " 'consequently': 13.81750955863044,\n",
       " 'consider': -2.765112078018567,\n",
       " 'considerably': -1.3833010954077654,\n",
       " 'considered': -13.81750955863044,\n",
       " 'consuming': -13.81750955863044,\n",
       " 'context': -13.81750955863044,\n",
       " 'contribution': -0.0,\n",
       " 'corpus': -13.81750955863044,\n",
       " 'correspondence': -1.3833010954077654,\n",
       " 'corrupt': -13.81750955863044,\n",
       " 'corrupted': -13.81750955863044,\n",
       " 'corrupting': -13.81750955863044,\n",
       " 'corruption': -13.81750955863044,\n",
       " 'cost': -13.81750955863044,\n",
       " 'count': -13.81750955863044,\n",
       " 'create': -13.81750955863044,\n",
       " 'creating': -13.81750955863044,\n",
       " 'crf': -13.81750955863044,\n",
       " 'crfsuite': -13.81750955863044,\n",
       " 'current': 13.81750955863044,\n",
       " 'data': -13.81750955863044,\n",
       " 'dataset': -2.191905425264702,\n",
       " 'datasets': 1.3833010954077654,\n",
       " 'decided': -13.81750955863044,\n",
       " 'deep': 13.81750955863044,\n",
       " 'define': -0.0,\n",
       " 'defined': -13.81750955863044,\n",
       " 'degrades': -13.81750955863044,\n",
       " 'demonstrate': 13.81750955863044,\n",
       " 'denoised': -13.81750955863044,\n",
       " 'denoiser': -13.81750955863044,\n",
       " 'denoising': -0.8092670144572764,\n",
       " 'denote': -13.81750955863044,\n",
       " 'denotes': -13.81750955863044,\n",
       " 'dense': -13.81750955863044,\n",
       " 'dependent': 13.81750955863044,\n",
       " 'depends': -13.81750955863044,\n",
       " 'describe': -13.81750955863044,\n",
       " 'designed': -1.3833010954077654,\n",
       " 'designing': 13.81750955863044,\n",
       " 'development': -13.81750955863044,\n",
       " 'dhillon': -13.81750955863044,\n",
       " 'diagonal': -13.81750955863044,\n",
       " 'dimension': -13.81750955863044,\n",
       " 'dimensional': -13.81750955863044,\n",
       " 'dimensionality': -13.81750955863044,\n",
       " 'directly': 13.81750955863044,\n",
       " 'distribution': -13.81750955863044,\n",
       " 'distributions': -13.81750955863044,\n",
       " 'divide': -13.81750955863044,\n",
       " 'divided': -13.81750955863044,\n",
       " 'documents': -0.0,\n",
       " 'domain': -0.9021040437790029,\n",
       " 'domains': -4.143212335940147,\n",
       " 'dominated': -13.81750955863044,\n",
       " 'dos': -13.81750955863044,\n",
       " 'dramatic': 13.81750955863044,\n",
       " 'draw': -13.81750955863044,\n",
       " 'drawn': -13.81750955863044,\n",
       " 'drop': -13.81750955863044,\n",
       " 'dropout': -1.8283917270672134,\n",
       " 'easier': 13.81750955863044,\n",
       " 'efficiency': 13.81750955863044,\n",
       " 'efficient': 13.81750955863044,\n",
       " 'efficiently': -13.81750955863044,\n",
       " 'elements': -13.81750955863044,\n",
       " 'eliminate': -13.81750955863044,\n",
       " 'eliminates': 13.81750955863044,\n",
       " 'emphasize': 13.81750955863044,\n",
       " 'english': -13.81750955863044,\n",
       " 'entire': -13.81750955863044,\n",
       " 'entries': -13.81750955863044,\n",
       " 'entry': -13.81750955863044,\n",
       " 'epochs': -13.81750955863044,\n",
       " 'equal': -13.81750955863044,\n",
       " 'equations': -2.191905425264702,\n",
       " 'equivalent': -13.81750955863044,\n",
       " 'estimate': -13.81750955863044,\n",
       " 'estimated': -13.81750955863044,\n",
       " 'estimation': -13.81750955863044,\n",
       " 'evaluate': -13.81750955863044,\n",
       " 'evaluating': 13.81750955863044,\n",
       " 'evaluation': -13.81750955863044,\n",
       " 'evolving': 13.81750955863044,\n",
       " 'exactly': -13.81750955863044,\n",
       " 'example': 13.81750955863044,\n",
       " 'expectation': -13.81750955863044,\n",
       " 'expectations': -13.81750955863044,\n",
       " 'experiments': -13.81750955863044,\n",
       " 'explain': -13.81750955863044,\n",
       " 'explicitly': 13.81750955863044,\n",
       " 'exploit': 1.3833010954077654,\n",
       " 'extend': -13.81750955863044,\n",
       " 'factor': -13.81750955863044,\n",
       " 'fail': 13.81750955863044,\n",
       " 'faster': 1.3833010954077654,\n",
       " 'feature': -1.1168795148738293,\n",
       " 'features': -3.362407660027699,\n",
       " 'feedback': 13.81750955863044,\n",
       " 'field': -13.81750955863044,\n",
       " 'fifty': -13.81750955863044,\n",
       " 'figure': -13.81750955863044,\n",
       " 'finally': -13.81750955863044,\n",
       " 'find': -0.0,\n",
       " 'fine': 13.81750955863044,\n",
       " 'fire': -13.81750955863044,\n",
       " 'firing': -13.81750955863044,\n",
       " 'focused': -13.81750955863044,\n",
       " 'focuses': -13.81750955863044,\n",
       " 'for': -13.81750955863044,\n",
       " 'form': -13.81750955863044,\n",
       " 'foundation': 13.81750955863044,\n",
       " 'framework': -13.81750955863044,\n",
       " 'free': -13.81750955863044,\n",
       " 'full': -13.81750955863044,\n",
       " 'functions': 13.81750955863044,\n",
       " 'fundamental': 13.81750955863044,\n",
       " 'fα': -13.81750955863044,\n",
       " 'fβ': -13.81750955863044,\n",
       " 'general': -13.81750955863044,\n",
       " 'generally': -13.81750955863044,\n",
       " 'generate': -13.81750955863044,\n",
       " 'grained': 13.81750955863044,\n",
       " 'graphs': -13.81750955863044,\n",
       " 'grow': 13.81750955863044,\n",
       " 'guo': -13.81750955863044,\n",
       " 'high': -13.81750955863044,\n",
       " 'historical': -2.4991159033707806,\n",
       " 'hmm': -13.81750955863044,\n",
       " 'hold': -13.81750955863044,\n",
       " 'hope': 13.81750955863044,\n",
       " 'huang': -13.81750955863044,\n",
       " 'hyper': -13.81750955863044,\n",
       " 'idea': 1.3833010954077654,\n",
       " 'ideas': 13.81750955863044,\n",
       " 'identical': -13.81750955863044,\n",
       " 'implementation': -1.3833010954077654,\n",
       " 'improvement': -13.81750955863044,\n",
       " 'improvements': 13.81750955863044,\n",
       " 'improving': -13.81750955863044,\n",
       " 'in': -2.765112078018567,\n",
       " 'include': -13.81750955863044,\n",
       " 'including': -13.81750955863044,\n",
       " 'incorporate': -13.81750955863044,\n",
       " 'incorporating': -13.81750955863044,\n",
       " 'increasing': -0.0,\n",
       " 'indicates': -13.81750955863044,\n",
       " 'individual': -0.0,\n",
       " 'induced': -13.81750955863044,\n",
       " 'input': -13.81750955863044,\n",
       " 'inputs': -13.81750955863044,\n",
       " 'insert': -13.81750955863044,\n",
       " 'instance': -1.3833010954077654,\n",
       " 'instances': -2.191905425264702,\n",
       " 'interest': -0.0,\n",
       " 'intuition': -13.81750955863044,\n",
       " 'intuitive': 13.81750955863044,\n",
       " 'inversion': -13.81750955863044,\n",
       " 'investigate': 13.81750955863044,\n",
       " 'involve': 13.81750955863044,\n",
       " 'involves': -13.81750955863044,\n",
       " 'is': -13.81750955863044,\n",
       " 'it': -13.81750955863044,\n",
       " 'italian': -13.81750955863044,\n",
       " 'known': -0.0,\n",
       " 'labeled': -1.3833010954077654,\n",
       " 'labeling': -1.3833010954077654,\n",
       " 'language': 1.3833010954077654,\n",
       " 'languages': -13.81750955863044,\n",
       " 'large': 1.3833010954077654,\n",
       " 'latent': -13.81750955863044,\n",
       " 'layer': -13.81750955863044,\n",
       " 'leads': -13.81750955863044,\n",
       " 'learn': -13.81750955863044,\n",
       " 'learned': 13.81750955863044,\n",
       " 'learning': -1.3833010954077654,\n",
       " 'level': -13.81750955863044,\n",
       " 'leverage': -13.81750955863044,\n",
       " 'linear': -13.81750955863044,\n",
       " 'log': -13.81750955863044,\n",
       " 'longer': -13.81750955863044,\n",
       " 'loss': -13.81750955863044,\n",
       " 'lossif': -13.81750955863044,\n",
       " 'low': -13.81750955863044,\n",
       " 'machine': -13.81750955863044,\n",
       " 'magnitude': -0.0,\n",
       " 'manually': -13.81750955863044,\n",
       " 'marginalization': -0.0,\n",
       " 'marginalize': -1.3833010954077654,\n",
       " 'marginalized': -0.0,\n",
       " 'matrices': -13.81750955863044,\n",
       " 'matrix': -13.81750955863044,\n",
       " 'mda': -2.191905425264702,\n",
       " 'mean': -13.81750955863044,\n",
       " 'media': 13.81750955863044,\n",
       " 'method': -13.81750955863044,\n",
       " 'methods': -2.765112078018567,\n",
       " 'middle': -13.81750955863044,\n",
       " 'minimizing': -13.81750955863044,\n",
       " 'model': -13.81750955863044,\n",
       " 'modern': -13.81750955863044,\n",
       " 'moon': -13.81750955863044,\n",
       " 'motivated': -13.81750955863044,\n",
       " 'multiple': -13.81750955863044,\n",
       " 'multiplication': -13.81750955863044,\n",
       " 'naive': -13.81750955863044,\n",
       " 'national': 13.81750955863044,\n",
       " 'natural': 13.81750955863044,\n",
       " 'ne': -13.81750955863044,\n",
       " 'networks': -13.81750955863044,\n",
       " 'neural': -13.81750955863044,\n",
       " 'newstext': -13.81750955863044,\n",
       " 'next': -13.81750955863044,\n",
       " 'nlp': -0.0,\n",
       " 'nogueira': -13.81750955863044,\n",
       " 'noise': -2.765112078018567,\n",
       " 'noising': 1.3833010954077654,\n",
       " 'nonlinearity': -13.81750955863044,\n",
       " 'normalization': -13.81750955863044,\n",
       " 'normalized': -13.81750955863044,\n",
       " 'note': -13.81750955863044,\n",
       " 'novel': -13.81750955863044,\n",
       " 'number': -0.0,\n",
       " 'observed': 13.81750955863044,\n",
       " 'obtaining': -0.0,\n",
       " 'occurence': -13.81750955863044,\n",
       " 'of': 0.5741998552952308,\n",
       " 'off': -13.81750955863044,\n",
       " 'older': -0.0,\n",
       " 'on': -13.81750955863044,\n",
       " 'optimization': -13.81750955863044,\n",
       " 'order': 13.81750955863044,\n",
       " 'orders': -13.81750955863044,\n",
       " 'ordinary': -13.81750955863044,\n",
       " 'original': -0.0,\n",
       " 'our': -13.81750955863044,\n",
       " 'outer': -13.81750955863044,\n",
       " 'outperforms': -13.81750955863044,\n",
       " 'output': -13.81750955863044,\n",
       " 'paper': 1.3833010954077654,\n",
       " 'parameter': -13.81750955863044,\n",
       " 'parameters': -13.81750955863044,\n",
       " 'parsing': 13.81750955863044,\n",
       " 'part': -0.0,\n",
       " 'passing': -13.81750955863044,\n",
       " 'pca': -13.81750955863044,\n",
       " 'pennacchiotti': -13.81750955863044,\n",
       " 'perform': -0.0,\n",
       " 'performance': -13.81750955863044,\n",
       " 'periods': -13.81750955863044,\n",
       " 'pilot': -13.81750955863044,\n",
       " 'pivot': -13.81750955863044,\n",
       " 'pivots': -13.81750955863044,\n",
       " 'portuguese': -1.3833010954077654,\n",
       " 'positive': -13.81750955863044,\n",
       " 'posterior': -13.81750955863044,\n",
       " 'predicting': -13.81750955863044,\n",
       " 'prediction': -13.81750955863044,\n",
       " 'presence': -13.81750955863044,\n",
       " 'presented': -13.81750955863044,\n",
       " 'presents': -13.81750955863044,\n",
       " 'preserved': -13.81750955863044,\n",
       " 'preserving': -13.81750955863044,\n",
       " 'previous': -1.3833010954077654,\n",
       " 'prior': -13.81750955863044,\n",
       " 'probability': -13.81750955863044,\n",
       " 'problem': -3.5718996842631983,\n",
       " 'process': 13.81750955863044,\n",
       " 'processes': -13.81750955863044,\n",
       " 'processing': 13.81750955863044,\n",
       " 'produced': 13.81750955863044,\n",
       " 'product': -13.81750955863044,\n",
       " 'progressively': -13.81750955863044,\n",
       " 'project': -13.81750955863044,\n",
       " 'projecting': -13.81750955863044,\n",
       " 'projection': -13.81750955863044,\n",
       " 'projections': -13.81750955863044,\n",
       " 'propose': -0.0,\n",
       " 'proposed': -1.3833010954077654,\n",
       " 'provide': 13.81750955863044,\n",
       " 'qα': -13.81750955863044,\n",
       " 'qβ': -13.81750955863044,\n",
       " 'random': -13.81750955863044,\n",
       " 'randomly': 2.191905425264702,\n",
       " 'rapidly': 13.81750955863044,\n",
       " 'rare': -13.81750955863044,\n",
       " 'ratio': -13.81750955863044,\n",
       " 'ratnaparkhi': -13.81750955863044,\n",
       " 'real': -13.81750955863044,\n",
       " 'realistic': 13.81750955863044,\n",
       " 'reasoning': -13.81750955863044,\n",
       " 'recall': -13.81750955863044,\n",
       " 'reconstruct': -13.81750955863044,\n",
       " 'reconstructing': -13.81750955863044,\n",
       " 'reconstruction': -13.81750955863044,\n",
       " 'reconstructions': -13.81750955863044,\n",
       " 'reduce': -13.81750955863044,\n",
       " 'refer': -13.81750955863044,\n",
       " 'regularization': -13.81750955863044,\n",
       " 'relevant': 13.81750955863044,\n",
       " 'relies': -13.81750955863044,\n",
       " 'rely': 13.81750955863044,\n",
       " 'remote': -13.81750955863044,\n",
       " 'remove': 13.81750955863044,\n",
       " 'representation': -1.3833010954077654,\n",
       " 'representations': -13.81750955863044,\n",
       " 'requires': -0.0,\n",
       " 'rescale': -13.81750955863044,\n",
       " 'rescaling': -13.81750955863044,\n",
       " 'resistant': 13.81750955863044,\n",
       " 'results': -13.81750955863044,\n",
       " 'review': -13.81750955863044,\n",
       " 'reviewers': 13.81750955863044,\n",
       " 'robust': 13.81750955863044,\n",
       " 'robustness': -13.81750955863044,\n",
       " 'roughly': -13.81750955863044,\n",
       " 'row': -13.81750955863044,\n",
       " 'sacrificing': 13.81750955863044,\n",
       " 'santos': -13.81750955863044,\n",
       " 'scaled': -13.81750955863044,\n",
       " 'scatter': -13.81750955863044,\n",
       " 'scenario': -13.81750955863044,\n",
       " 'scheme': -13.81750955863044,\n",
       " 'science': 13.81750955863044,\n",
       " 'scl': -3.2093102426640705,\n",
       " 'scramble': -13.81750955863044,\n",
       " 'scrambling': -2.191905425264702,\n",
       " 'secondary': 13.81750955863044,\n",
       " 'select': -13.81750955863044,\n",
       " 'selecting': -13.81750955863044,\n",
       " 'selects': 13.81750955863044,\n",
       " 'sentiment': 13.81750955863044,\n",
       " 'sequence': -1.3833010954077654,\n",
       " 'set': -13.81750955863044,\n",
       " 'sets': -13.81750955863044,\n",
       " 'setting': -13.81750955863044,\n",
       " 'settings': -13.81750955863044,\n",
       " 'sgd': -13.81750955863044,\n",
       " 'showing': 13.81750955863044,\n",
       " 'simple': -13.81750955863044,\n",
       " 'simpler': 13.81750955863044,\n",
       " 'simplicity': 13.81750955863044,\n",
       " 'single': -0.0,\n",
       " 'slowed': -13.81750955863044,\n",
       " 'slower': -13.81750955863044,\n",
       " 'smaller': -13.81750955863044,\n",
       " 'social': 13.81750955863044,\n",
       " 'solution': -0.0,\n",
       " 'solutions': -13.81750955863044,\n",
       " 'solve': -13.81750955863044,\n",
       " 'solving': -1.3833010954077654,\n",
       " 'source': -13.81750955863044,\n",
       " 'space': -0.0,\n",
       " 'spaces': 2.191905425264702,\n",
       " 'specific': -13.81750955863044,\n",
       " 'spectral': -13.81750955863044,\n",
       " 'speech': -0.0,\n",
       " 'speedups': 13.81750955863044,\n",
       " 'spirit': 13.81750955863044,\n",
       " 'split': -13.81750955863044,\n",
       " 'square': -13.81750955863044,\n",
       " 'squared': -13.81750955863044,\n",
       " 'stacking': -13.81750955863044,\n",
       " 'stale': 13.81750955863044,\n",
       " 'standard': -3.2093102426640705,\n",
       " 'statistics': -13.81750955863044,\n",
       " 'step': 13.81750955863044,\n",
       " 'structural': -1.3833010954077654,\n",
       " 'structure': 2.191905425264702,\n",
       " 'structured': -0.0,\n",
       " 'styles': 13.81750955863044,\n",
       " 'sub': -13.81750955863044,\n",
       " 'subset': -13.81750955863044,\n",
       " 'subsets': -13.81750955863044,\n",
       " 'substantial': 13.81750955863044,\n",
       " 'suffix': 13.81750955863044,\n",
       " 'sum': -13.81750955863044,\n",
       " 'supervised': -13.81750955863044,\n",
       " 'supported': 13.81750955863044,\n",
       " 'supports': -13.81750955863044,\n",
       " 'survives': -13.81750955863044,\n",
       " 'syntactic': -0.0,\n",
       " 'synthetic': 13.81750955863044,\n",
       " 'system': -1.3833010954077654,\n",
       " 'systems': -0.0,\n",
       " 's𝐖s': -13.81750955863044,\n",
       " 'table': -13.81750955863044,\n",
       " 'tackle': -13.81750955863044,\n",
       " 'tagged': -13.81750955863044,\n",
       " 'tagger': -13.81750955863044,\n",
       " 'tagging': -0.8092670144572764,\n",
       " 'tags': -13.81750955863044,\n",
       " 'tanh': -13.81750955863044,\n",
       " 'target': -13.81750955863044,\n",
       " 'task': -13.81750955863044,\n",
       " 'tasks': 0.8092670144572764,\n",
       " 'technique': -13.81750955863044,\n",
       " 'techniques': -0.0,\n",
       " 'template': -0.4453889214302198,\n",
       " 'templates': -3.2093102426640705,\n",
       " 'temporally': -13.81750955863044,\n",
       " 'tend': 13.81750955863044,\n",
       " 'terms': -13.81750955863044,\n",
       " 'test': -13.81750955863044,\n",
       " 'testing': -13.81750955863044,\n",
       " 'text': -13.81750955863044,\n",
       " 'texts': 1.3833010954077654,\n",
       " 'the': -13.81750955863044,\n",
       " 'third': -13.81750955863044,\n",
       " 'thousands': 13.81750955863044,\n",
       " 'three': -13.81750955863044,\n",
       " 'time': -13.81750955863044,\n",
       " 'times': -13.81750955863044,\n",
       " 'token': -13.81750955863044,\n",
       " 'topic': -13.81750955863044,\n",
       " 'total': -13.81750955863044,\n",
       " 'toutanova': 13.81750955863044,\n",
       " 'train': -13.81750955863044,\n",
       " 'trained': 13.81750955863044,\n",
       " 'training': -3.8781683679698267,\n",
       " 'transfer': -13.81750955863044,\n",
       " 'transform': 13.81750955863044,\n",
       " 'transformed': -13.81750955863044,\n",
       " 'transition': -13.81750955863044,\n",
       " 'treated': -13.81750955863044,\n",
       " 'trend': -13.81750955863044,\n",
       " 'tunable': -13.81750955863044,\n",
       " 'tune': -13.81750955863044,\n",
       " 'tycho': -13.81750955863044,\n",
       " 'type': -13.81750955863044,\n",
       " 'types': -2.191905425264702,\n",
       " 'unchanged': -13.81750955863044,\n",
       " 'uncorrupted': -13.81750955863044,\n",
       " 'uniform': -13.81750955863044,\n",
       " 'unlike': -13.81750955863044,\n",
       " 'unsupervised': 1.3833010954077654,\n",
       " 'valued': -13.81750955863044,\n",
       " 'values': -13.81750955863044,\n",
       " 'variance': -13.81750955863044,\n",
       " 'vector': -13.81750955863044,\n",
       " 'version': -13.81750955863044,\n",
       " 'versions': -13.81750955863044,\n",
       " 'view': -13.81750955863044,\n",
       " 'we': -2.191905425264702,\n",
       " 'weight': -13.81750955863044,\n",
       " 'well': -0.0,\n",
       " 'will': -4.143212335940147,\n",
       " 'word': -0.5741998552952308,\n",
       " 'words': -0.0,\n",
       " 'work': -1.8283917270672134,\n",
       " 'write': -13.81750955863044,\n",
       " 'writing': 13.81750955863044,\n",
       " 'written': -13.81750955863044,\n",
       " 'xiao': -13.81750955863044,\n",
       " 'yates': -13.81750955863044,\n",
       " 'year': -13.81750955863044,\n",
       " 'years': 13.81750955863044,\n",
       " 'yt': -13.81750955863044,\n",
       " 'zanzotto': -13.81750955863044,\n",
       " 'ℝd': -13.81750955863044,\n",
       " '𝐐α': -13.81750955863044,\n",
       " '𝐖s': -13.81750955863044,\n",
       " '𝐖𝐗': -13.81750955863044,\n",
       " '𝐗s': -13.81750955863044,\n",
       " '𝐗𝐗': -13.81750955863044,\n",
       " '𝐱i': -13.81750955863044,\n",
       " '𝐱n': -13.81750955863044}"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dict(llr_f(inputs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = vectorizer.fit_transform(inputs.data).toarray()\n",
    "Y = np.array(inputs.target)\n",
    "Y = LabelBinarizer().fit_transform(Y)\n",
    "if Y.shape[1] == 1:\n",
    "    Y = np.append(1 - Y, Y, axis=1)\n",
    "\n",
    "prob = np.dot(Y.T, X) / np.sum(X, axis=0) + 1e-3   # n_classes * n_features\n",
    "prob = prob / np.sum(prob, axis=0) \n",
    "sign = -2 * (np.log(prob[0]) - np.log(prob[1]))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([559.90975328, 553.69314718, 559.90975328, 559.90975328,\n",
       "       553.69314718, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 554.09761478, 559.90975328,\n",
       "       559.90975328, 553.88687557, 559.90975328, 559.90975328,\n",
       "       559.90975328, 554.20264302, 553.69314718, 559.90975328,\n",
       "       559.90975328, 554.09761478, 559.90975328, 554.09761478,\n",
       "       559.90975328, 553.69314718, 553.69314718, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 553.91579185,\n",
       "       559.90975328, 554.09761478, 553.69314718, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 553.84696525, 553.84696525, 559.90975328,\n",
       "       559.90975328, 553.69314718, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 553.91579185, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 554.60644837,\n",
       "       554.09761478, 559.90975328, 559.90975328, 559.90975328,\n",
       "       553.69314718, 559.90975328, 554.09761478, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 554.38430034,\n",
       "       554.09761478, 559.90975328, 559.90975328, 553.69314718,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 553.91579185, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       554.09761478, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 553.69314718, 553.94389148, 555.19026284,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 554.25126708, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 554.38430034, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 554.09761478, 559.90975328, 559.90975328,\n",
       "       559.90975328, 554.09761478, 554.01085269, 554.85191639,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 553.69314718, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 554.50158549, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 554.09761478,\n",
       "       559.90975328, 559.90975328, 554.09761478, 559.90975328,\n",
       "       559.90975328, 559.90975328, 554.60644837, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 553.69314718,\n",
       "       559.90975328, 553.69314718, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 554.09761478, 554.38430034,\n",
       "       553.69314718, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 553.69314718, 554.09761478,\n",
       "       554.09761478, 554.09761478, 559.90975328, 554.09761478,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 554.09761478, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 553.69314718,\n",
       "       559.90975328, 553.69314718, 554.09761478, 553.69314718,\n",
       "       559.90975328, 559.90975328, 554.38430034, 559.90975328,\n",
       "       559.90975328, 559.90975328, 554.60644837, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 553.69314718,\n",
       "       559.90975328, 554.60644837, 554.09761478, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       553.69314718, 559.90975328, 553.69314718, 559.90975328,\n",
       "       553.84696525, 559.90975328, 553.69314718, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       553.69314718, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 554.09761478, 559.90975328, 559.90975328,\n",
       "       559.90975328, 553.69314718, 559.90975328, 559.90975328,\n",
       "       559.90975328, 553.69314718, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 554.09761478,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 554.09761478, 559.90975328, 559.90975328,\n",
       "       554.94093254, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 553.69314718,\n",
       "       554.09761478, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 554.38430034, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 554.09761478, 559.90975328,\n",
       "       553.69314718, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 554.7877754 , 559.90975328,\n",
       "       554.38430034, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 554.09761478, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       553.69314718, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 553.69314718, 559.90975328, 559.90975328,\n",
       "       554.09761478, 559.90975328, 553.69314718, 554.38430034,\n",
       "       559.90975328, 559.90975328, 553.69314718, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 554.7877754 , 559.90975328,\n",
       "       559.90975328, 554.09761478, 554.38430034, 553.69314718,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 553.69314718,\n",
       "       559.90975328, 554.09761478, 553.69314718, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       553.91579185, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 553.91579185, 559.90975328, 553.69314718,\n",
       "       553.81068075, 554.7877754 , 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       554.09761478, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 555.07347137, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       554.38430034, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 554.09761478, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 554.38430034, 559.90975328, 553.69314718,\n",
       "       555.19026284, 553.84696525, 553.69314718, 554.25126708,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328, 559.90975328, 559.90975328, 559.90975328,\n",
       "       559.90975328])"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.max(np.log(prob), axis=0).shape - np.min(np.log(prob), axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[9.98003992e-04, 5.00000000e-01, 9.99001996e-01, ...,\n",
       "        9.99001996e-01, 9.99001996e-01, 9.99001996e-01],\n",
       "       [9.99001996e-01, 5.00000000e-01, 9.98003992e-04, ...,\n",
       "        9.98003992e-04, 9.98003992e-04, 9.98003992e-04]])"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[9.98003992e-04, 5.00000000e-01, 9.99001996e-01, ...,\n",
       "        9.99001996e-01, 9.99001996e-01, 9.99001996e-01],\n",
       "       [9.99001996e-01, 5.00000000e-01, 9.98003992e-04, ...,\n",
       "        9.98003992e-04, 9.98003992e-04, 9.98003992e-04]])"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.0986122886681096"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.log(0.2) - np.log(0.6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob = prob / np.sum(X, axis=0) + 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.00000000e-02, 2.55000000e-01, 1.01000000e+00, ...,\n",
       "        1.01000000e+00, 8.41666667e-02, 5.05000000e-01],\n",
       "       [1.01000000e+00, 2.55000000e-01, 1.00000000e-02, ...,\n",
       "        1.00000000e-02, 8.33333333e-04, 5.00000000e-03]])"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prob / np.sum(X, axis=0) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-4.605170185988091"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.log(0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = item.transform()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vectorizer = CountVectorizer(lowercase=False)\n",
    "\n",
    "from sklearn.preprocessing import LabelBinarizer\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "inputs = item.transform()\n",
    "X = vectorizer.fit_transform(inputs.data)\n",
    "Y = np.array(inputs.target)\n",
    "Y = LabelBinarizer().fit_transform(Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(102, 553)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "if Y.shape[1] == 1:\n",
    "        Y = np.append(1 - Y, Y, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(102, 2)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "ob = np.dot(Y.T, X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 102)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ob.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "if Y.shape[1] == 1:\n",
    "        Y = np.append(1 - Y, Y, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(102, 2)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob = np.dot(Y.T, X.toarray())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 553)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prob.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([[1, 2, 3], [2, 3,  4\n",
    "                         ]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2, 3],\n",
       "       [2, 3, 4]])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.33333333, 0.4       , 0.42857143],\n",
       "       [0.66666667, 0.6       , 0.57142857]])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a / a.sum(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "b = a / a.sum(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.38629436, 0.81093022, 0.57536414])"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "-2 * (np.log(b[0]) - np.log(b[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.3862943611198908"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "-2 * (np.log(0.33) - np.log(0.66))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0,  1,  1, ...,  1, 12,  2],\n",
       "       [ 1,  1,  0, ...,  0,  0,  0]], dtype=int64)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0. , 0.5, 1. , ..., 1. , 1. , 1. ],\n",
       "       [1. , 0.5, 0. , ..., 0. , 0. , 0. ]])"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prob / prob.sum(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0,  1,  1, ...,  1, 12,  2],\n",
       "       [ 1,  1,  0, ...,  0,  0,  0]], dtype=int64)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [17],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [10],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 6],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 4],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 5],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 4],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 7],\n",
       "       [ 7],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 4],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 5],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 4],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 5],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 2],\n",
       "       [ 5],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 6],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [12],\n",
       "       [ 4],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [10],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 6],\n",
       "       [ 1],\n",
       "       [ 4],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 2],\n",
       "       [ 2],\n",
       "       [ 2],\n",
       "       [ 2],\n",
       "       [18],\n",
       "       [ 9],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [21],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 2],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 4],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 5],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [33],\n",
       "       [32],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 9],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 5],\n",
       "       [ 3],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 4],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 3],\n",
       "       [ 3],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 9],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 3],\n",
       "       [ 6],\n",
       "       [ 2],\n",
       "       [13],\n",
       "       [ 8],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 5],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 4],\n",
       "       [ 1],\n",
       "       [25],\n",
       "       [ 6],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 4],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 7],\n",
       "       [ 3],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 6],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 5],\n",
       "       [ 1],\n",
       "       [ 4],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 5],\n",
       "       [ 2],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 8],\n",
       "       [ 7],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 2],\n",
       "       [ 2],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 3],\n",
       "       [ 2],\n",
       "       [ 4],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 6],\n",
       "       [ 2],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 6],\n",
       "       [ 1],\n",
       "       [ 4],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 6],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 4],\n",
       "       [ 4],\n",
       "       [ 2],\n",
       "       [ 3],\n",
       "       [ 3],\n",
       "       [ 2],\n",
       "       [ 4],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 4],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 6],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 4],\n",
       "       [14],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 4],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 5],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 5],\n",
       "       [ 1],\n",
       "       [ 5],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 9],\n",
       "       [ 6],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 2],\n",
       "       [ 2],\n",
       "       [ 6],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 8],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 2],\n",
       "       [ 2],\n",
       "       [ 4],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 5],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 4],\n",
       "       [ 1],\n",
       "       [ 4],\n",
       "       [ 9],\n",
       "       [ 7],\n",
       "       [ 2],\n",
       "       [ 7],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 2],\n",
       "       [ 1],\n",
       "       [ 3],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [ 1],\n",
       "       [12],\n",
       "       [ 2]], dtype=int64)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prob.sum(axis=0).reshape(-1, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(553,)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prob.sum(axis=0).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_count = X.sum(axis=0).reshape(-1, 1)\n",
    "class_prob = Y.mean(axis=0).reshape(1, -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "shapes (2,1) and (553,1) not aligned: 1 (dim 1) != 553 (dim 0)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-16-16b932004c2f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mexpected\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mclass_prob\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeature_count\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m: shapes (2,1) and (553,1) not aligned: 1 (dim 1) != 553 (dim 0)"
     ]
    }
   ],
   "source": [
    "expected = np.dot(class_prob.T, feature_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.79411765, 0.20588235])"
      ]
     },
     "execution_count": 257,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y.mean(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.sum(axis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(102, 553)"
      ]
     },
     "execution_count": 250,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1]])"
      ]
     },
     "execution_count": 228,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Input(data=['unsupervised domain adaptation fundamental problem natural language processing hope apply systems datasets annotations', 'relevant labeled datasets stale comparison rapidly evolving social media writing styles increasing interest natural language processing historical texts', 'number approaches domain adaptation proposed tend emphasize bag-of-words features classification tasks sentiment analysis', 'consequently approaches rely instance large number active features fail exploit structured feature spaces characterize syntactic tasks sequence labeling parsing .as will substantial efficiency improvements designing domain adaptation methods learning structured feature spaces', 'build work deep learning community denoising autoencoders trained remove synthetic noise observed instances', 'autoencoder transform original feature space representation dependent individual feature robust domains', 'chen al', 'autoencoders learned noising process analytically marginalized idea spirit feature noising', 'marginalized denoising autoencoder mda considerably faster original denoising autoencoder requires solving system equations grow large realistic nlp tasks involve features.in paper investigate noising functions explicitly designed structured feature spaces common nlp', 'example part-of-speech tagging toutanova al', 'define feature “ templates ” current word previous word suffix current word', 'feature template thousands binary features', 'exploit structure propose alternative noising techniques 1 feature scrambling randomly chooses feature template randomly selects alternative template 2 structured dropout randomly eliminates single feature template', 'marginalize types noise find solution structured dropout simpler efficient mda approach chen al', 'consider feature structure.we apply ideas fine-grained part-of-speech tagging dataset portuguese texts years 1502 1836 training texts evaluating older documents', 'structure-aware domain adaptation algorithms perform well standard dropout — better well-known structural correspondence learning scl algorithm — structured dropout order-of-magnitude faster', 'secondary contribution paper demonstrate applicability unsupervised domain adaptation syntactic analysis historical texts', 'describe denoising autoencoder application domain adaptation analytic marginalization noise', 'three versions marginalized denoising autoencoders mda incorporating types noise including noising processes designed structured features.assume instances 𝐱 … 𝐱n drawn source target domains', 'will “ corrupt ” instances adding types noise denote corrupted version 𝐱i 𝐱~i', 'single-layer denoising autoencoders reconstruct corrupted inputs projection matrix 𝐖 ℝd→ℝd estimated minimizing squared reconstruction lossif write 𝐗=𝐱 … 𝐱n∈ℝd×n write corrupted version 𝐗~ loss written asin case well-known closed-form solution ordinary square problem 𝐐=𝐗~\\u2062𝐗~⊤ 𝐏=𝐗\\u2062𝐗~⊤', 'obtaining weight matrix 𝐖 insert nonlinearity output denoiser tanh\\u2061𝐖𝐗', 'apply stacking passing vector autoencoder', 'pilot experiments slowed estimation accuracy include it.structured prediction tasks features simple bag-of-words representation performance relies rare features', 'naive implementation denoising approach 𝐏 𝐐 will dense matrices dimensionality d×d roughly elements experiments', 'solve problem chen al', 'propose set pivot features train autoencoder reconstruct pivots full set features', 'corrupted input divided subsets 𝐱~i=𝐱~i⊤ … 𝐱~is⊤⊤', 'projection matrix 𝐖s subset reconstructing pivot features features subset sum reconstructions features tanh\\u2061∑s=s𝐖s\\u2062𝐗s.in standard denoising autoencoder generate multiple versions corrupted data 𝐗~ reduce variance solution', 'chen al', 'marginalize noise analytically computing expectations 𝐏 𝐐 computingwhere e\\u2062𝐏=∑i=ne\\u2062𝐱i\\u2062𝐱~i⊤ e\\u2062𝐐=∑i=ne\\u2062𝐱~i\\u2062𝐱~i⊤', 'equivalent corrupting data m→∞ times', 'computation expectations depends type noise.chen al', 'dropout noise domain adaptation review', 'describe novel types noise designed structured feature spaces explain marginalized efficiently compute 𝐖.in dropout noise feature set probability', 'define scatter matrix uncorrupted input 𝐒=𝐗𝐗⊤ solutions dropout noise areandwhere α β features', 'form solutions computing 𝐖 requires solving system equations equal number features naive implementation smaller systems equations high-dimensional version', 'note tunable parameter type noise.in nlp settings feature templates previous-word middle-word next-word feature template firing token', 'exploit structure alternative dropout scheme token choose exactly feature template features consider token transition feature templates ⟨yt yt-⟩ considered dropout', 'assuming feature templates noise leads simple solutions marginalized matrices e\\u2062𝐏 e\\u2062𝐐 e\\u2062𝐏 scaled version scatter matrix instance 𝐱~ exactly /k chance individual feature survives dropout', 'e\\u2062𝐐 diagonal off-diagonal entry e\\u2062𝐐α β α β will drop instance', 'view projection matrix 𝐖 row-normalized version scatter matrix 𝐒', 'contribution β reconstruction α equal co-occurence count α β divided count β.unlike standard dropout free hyper-parameters tune structured dropout', 'e\\u2062𝐐 diagonal matrix eliminate cost matrix inversion solving system linear equations', 'extend mda high dimensional data longer divide corrupted input 𝐱~ subsets.e\\u2062𝐏 matrix number pivots.for intuition consider standard feature dropout p=k-k', 'will structured dropout matrix e\\u2062𝐏 identical e\\u2062𝐐 off-diagonal elements scaled -p large', 'including elements standard dropout considerably slower experiments.a third alternative “ scramble ” features randomly selecting alternative features template', 'feature α belonging template probability will draw noise feature β belonging distribution', 'work uniform distribution qβ=|f|', 'solutions will hold scrambling distributions mean-preserving distributions.again analytically marginalize noise', 'recall e\\u2062𝐐=∑i=ne\\u2062𝐱~i\\u2062𝐱~i⊤', 'off-diagonal entry matrix 𝐱~\\u2062𝐱~⊤ involves features α β belonging templates fα≠fβ values 𝐱i α denotes feature α 𝐱i 𝐱i α\\u2062𝐱i β features unchanged probability -p', 'features chosen noise features probability p\\u2062qα\\u2062qβ.𝐱i α 𝐱i β feature unchanged chosen noise feature probability p\\u2062-p\\u2062qβ p\\u2062-p\\u2062qα.the diagonal entries values probability -p p\\u2062qα', 'entries will feature belonging template will fire 𝐱i', 'reasoning compute expectation 𝐏', 'probability -p original features preserved add outer-product 𝐱i\\u2062𝐱i⊤ probability add outer-product 𝐱i\\u2062q⊤', 'e\\u2062𝐏 computed sum terms', 'compare methods historical portuguese part-of-speech tagging creating domains historical epochs.we tycho brahe corpus evaluate methods', 'corpus total 1,480,528 manually tagged', 'set 383 tags composed texts historical portuguese 1502 1836', 'divide texts fifty-year periods create domains', 'table presents statistics datasets', 'hold 5 data development data tune parameters', 'domains 1800-1849 1750-1849 treated source domains domains target domains', 'scenario motivated training tagger modern newstext corpus applying historical documents.we conditional random field tagger choosing crfsuite supports arbitrary real valued features sgd optimization', 'work nogueira dos santos al', 'dataset apply feature set ratnaparkhi', 'feature templates features total', 'blitzer al', 'consider pivot features appear times domains', 'leads total pivot features experiments.we compare mda three alternative approaches', 'refer baseline training crf tagger source domain testing target domain base features', 'include pca project entire dataset low-dimensional sub-space including original features', 'finally compare structural correspondence learning scl blitzer al. 2006 feature learning algorithm', 'cases include entire dataset compute feature projections conducted experiments test training data feature projections results.all hyper-parameters decided development data training set', 'low dimension pca', 'blitzer perform feature centering/normalization well rescaling scl', 'best parameters scl dimensionality k= rescale factor α= original paper', 'mda best corruption level p=0.9 dropout noise p=0.1 scrambling noise', 'structured dropout noise free hyper-parameters.table presents domain adaptation tasks', 'compute transfer ratio defined adaptation accuracybaseline accuracy figure', 'generally positive trend graphs indicates adaptation progressively select test sets temporally remote training data.in general mda outperforms scl pca improvement base features', 'noising approaches mda', 'structured dropout orders magnitude faster alternatives table', 'scrambling noise time-consuming cost dominated matrix multiplication', 'previous work domain adaptation focused supervised setting labeled data target domain', 'work focuses unsupervised domain adaptation labeled data target domain', 'representation learning methods proposed solve problem', 'structural correspondence learning scl induced representation based task predicting presence pivot features', 'autoencoders apply idea denoised instances latent representation', 'context denoising autoencoders focused dropout noise applied general technique improving robustness machine learning neural networks .on specific problem sequence labeling xiao guo proposed supervised domain adaptation method log-bilinear language adaptation model', 'dhillon al', 'presented spectral method estimate low dimensional context-specific word representations sequence labeling', 'huang yates hmm model learn latent representations leverage posterior regularization framework incorporate specific biases', 'methods approach standard crf transformed features.our evaluation concerns syntactic analysis historical text topic increasing interest nlp', 'pennacchiotti zanzotto find part-of-speech tagging degrades considerably applied corpus historical italian', 'moon baldridge tackle challenging problem tagging middle english techniques projecting syntactic annotations languages', 'prior work tycho brahe corpus applied supervised learning random split test training data consider domain adaptation problem training data testing older historical text', 'denoising autoencoders provide intuitive solution domain adaptation transform features representation resistant noise characterize domain adaptation process', 'original implementation idea produced noise directly work dropout noise analytically marginalized', 'step simplicity showing structured dropout marginalization easier obtaining dramatic speedups sacrificing accuracy reviewers feedback', 'supported national science foundation award 1349837'], target=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1])"
      ]
     },
     "execution_count": 223,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'rouge-1': {'f': 0.23140495570247935,\n",
       "   'p': 0.1414141414141414,\n",
       "   'r': 0.6363636363636364},\n",
       "  'rouge-2': {'f': 0.043343650808116765,\n",
       "   'p': 0.02527075812274368,\n",
       "   'r': 0.15217391304347827},\n",
       "  'rouge-l': {'f': 0.11009004563932587,\n",
       "   'p': 0.10606060606060606,\n",
       "   'r': 0.4772727272727273}}]"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = \" \".join(map(lambda s: \" \".join(s), item.summarize(\"chi2\")[:]))\n",
    "rouge_score(truth, s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "230"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(s.split(\" \"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['autoencoders',\n",
       "  'learned',\n",
       "  'noising',\n",
       "  'process',\n",
       "  'analytically',\n",
       "  'marginalized',\n",
       "  'idea',\n",
       "  'spirit',\n",
       "  'feature',\n",
       "  'noising'],\n",
       " ['autoencoder',\n",
       "  'transform',\n",
       "  'original',\n",
       "  'feature',\n",
       "  'space',\n",
       "  'representation',\n",
       "  'dependent',\n",
       "  'individual',\n",
       "  'feature',\n",
       "  'robust',\n",
       "  'domains'],\n",
       " ['secondary',\n",
       "  'contribution',\n",
       "  'paper',\n",
       "  'demonstrate',\n",
       "  'applicability',\n",
       "  'unsupervised',\n",
       "  'domain',\n",
       "  'adaptation',\n",
       "  'syntactic',\n",
       "  'analysis',\n",
       "  'historical',\n",
       "  'texts.denoising',\n",
       "  'autoencoders',\n",
       "  'provide',\n",
       "  'intuitive',\n",
       "  'solution',\n",
       "  'domain',\n",
       "  'adaptation',\n",
       "  'transform',\n",
       "  'features',\n",
       "  'representation',\n",
       "  'resistant',\n",
       "  'noise',\n",
       "  'characterize',\n",
       "  'domain',\n",
       "  'adaptation',\n",
       "  'process'],\n",
       " ['exploit',\n",
       "  'structure',\n",
       "  'propose',\n",
       "  'alternative',\n",
       "  'noising',\n",
       "  'techniques',\n",
       "  '1',\n",
       "  'feature',\n",
       "  'scrambling',\n",
       "  'randomly',\n",
       "  'chooses',\n",
       "  'feature',\n",
       "  'template',\n",
       "  'randomly',\n",
       "  'selects',\n",
       "  'alternative',\n",
       "  'template',\n",
       "  '2',\n",
       "  'structured',\n",
       "  'dropout',\n",
       "  'randomly',\n",
       "  'eliminates',\n",
       "  'single',\n",
       "  'feature',\n",
       "  'template'],\n",
       " ['marginalize',\n",
       "  'types',\n",
       "  'noise',\n",
       "  'find',\n",
       "  'solution',\n",
       "  'structured',\n",
       "  'dropout',\n",
       "  'simpler',\n",
       "  'efficient',\n",
       "  'mda',\n",
       "  'approach',\n",
       "  'chen',\n",
       "  'al'],\n",
       " ['structure-aware',\n",
       "  'domain',\n",
       "  'adaptation',\n",
       "  'algorithms',\n",
       "  'perform',\n",
       "  'well',\n",
       "  'standard',\n",
       "  'dropout',\n",
       "  '—',\n",
       "  'better',\n",
       "  'well-known',\n",
       "  'structural',\n",
       "  'correspondence',\n",
       "  'learning',\n",
       "  'scl',\n",
       "  'algorithm',\n",
       "  '—',\n",
       "  'structured',\n",
       "  'dropout',\n",
       "  'order-of-magnitude',\n",
       "  'faster'],\n",
       " ['original',\n",
       "  'implementation',\n",
       "  'idea',\n",
       "  'produced',\n",
       "  'noise',\n",
       "  'directly',\n",
       "  'work',\n",
       "  'dropout',\n",
       "  'noise',\n",
       "  'analytically',\n",
       "  'marginalized'],\n",
       " ['feature', 'template', 'thousands', 'binary', 'features'],\n",
       " ['chen', 'al']]"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item.summarize(\"idf\")[]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['supported', 'national', 'science', 'foundation', 'award', '1349837'],\n",
       " ['relevant',\n",
       "  'labeled',\n",
       "  'datasets',\n",
       "  'stale',\n",
       "  'comparison',\n",
       "  'rapidly',\n",
       "  'evolving',\n",
       "  'social',\n",
       "  'media',\n",
       "  'writing',\n",
       "  'styles',\n",
       "  'increasing',\n",
       "  'interest',\n",
       "  'natural',\n",
       "  'language',\n",
       "  'processing',\n",
       "  'historical',\n",
       "  'texts'],\n",
       " ['step',\n",
       "  'simplicity',\n",
       "  'showing',\n",
       "  'structured',\n",
       "  'dropout',\n",
       "  'marginalization',\n",
       "  'easier',\n",
       "  'obtaining',\n",
       "  'dramatic',\n",
       "  'speedups',\n",
       "  'sacrificing',\n",
       "  'accuracy',\n",
       "  'reviewers',\n",
       "  'feedback'],\n",
       " ['build',\n",
       "  'work',\n",
       "  'deep',\n",
       "  'learning',\n",
       "  'community',\n",
       "  'denoising',\n",
       "  'autoencoders',\n",
       "  'trained',\n",
       "  'remove',\n",
       "  'synthetic',\n",
       "  'noise',\n",
       "  'observed',\n",
       "  'instances'],\n",
       " ['example', 'part-of-speech', 'tagging', 'toutanova', 'al'],\n",
       " ['define',\n",
       "  'feature',\n",
       "  '“',\n",
       "  'templates',\n",
       "  '”',\n",
       "  'current',\n",
       "  'word',\n",
       "  'previous',\n",
       "  'word',\n",
       "  'suffix',\n",
       "  'current',\n",
       "  'word'],\n",
       " ['consider',\n",
       "  'feature',\n",
       "  'structure.we',\n",
       "  'apply',\n",
       "  'ideas',\n",
       "  'fine-grained',\n",
       "  'part-of-speech',\n",
       "  'tagging',\n",
       "  'dataset',\n",
       "  'portuguese',\n",
       "  'texts',\n",
       "  'years',\n",
       "  '1502',\n",
       "  '1836',\n",
       "  'training',\n",
       "  'texts',\n",
       "  'evaluating',\n",
       "  'older',\n",
       "  'documents'],\n",
       " ['unsupervised',\n",
       "  'domain',\n",
       "  'adaptation',\n",
       "  'fundamental',\n",
       "  'problem',\n",
       "  'natural',\n",
       "  'language',\n",
       "  'processing',\n",
       "  'hope',\n",
       "  'apply',\n",
       "  'systems',\n",
       "  'datasets',\n",
       "  'annotations'],\n",
       " ['marginalized',\n",
       "  'denoising',\n",
       "  'autoencoder',\n",
       "  'mda',\n",
       "  'considerably',\n",
       "  'faster',\n",
       "  'original',\n",
       "  'denoising',\n",
       "  'autoencoder',\n",
       "  'requires',\n",
       "  'solving',\n",
       "  'system',\n",
       "  'equations',\n",
       "  'grow',\n",
       "  'large',\n",
       "  'realistic',\n",
       "  'nlp',\n",
       "  'tasks',\n",
       "  'involve',\n",
       "  'features.in',\n",
       "  'paper',\n",
       "  'investigate',\n",
       "  'noising',\n",
       "  'functions',\n",
       "  'explicitly',\n",
       "  'designed',\n",
       "  'structured',\n",
       "  'feature',\n",
       "  'spaces',\n",
       "  'common',\n",
       "  'nlp'],\n",
       " ['number',\n",
       "  'approaches',\n",
       "  'domain',\n",
       "  'adaptation',\n",
       "  'proposed',\n",
       "  'tend',\n",
       "  'emphasize',\n",
       "  'bag-of-words',\n",
       "  'features',\n",
       "  'classification',\n",
       "  'tasks',\n",
       "  'sentiment',\n",
       "  'analysis']]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item.summarize(\"idf\")[:-10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'rouge-1': {'f': 0.23437499548828133,\n",
       "   'p': 0.17857142857142858,\n",
       "   'r': 0.3409090909090909},\n",
       "  'rouge-2': {'f': 0.0370370329705842,\n",
       "   'p': 0.02586206896551724,\n",
       "   'r': 0.06521739130434782},\n",
       "  'rouge-l': {'f': 0.15917673715950592,\n",
       "   'p': 0.14285714285714285,\n",
       "   'r': 0.2727272727272727}}]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = \" \".join(map(lambda s: \" \".join(s), item.summarize(\"idf\")[11:20]))\n",
    "rouge_score(truth, s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'rouge-1': {'f': 0.27544909791530714,\n",
       "   'p': 0.18699186991869918,\n",
       "   'r': 0.5227272727272727},\n",
       "  'rouge-2': {'f': 0.0756756719392259,\n",
       "   'p': 0.050359712230215826,\n",
       "   'r': 0.15217391304347827},\n",
       "  'rouge-l': {'f': 0.13153560723700325,\n",
       "   'p': 0.12195121951219512,\n",
       "   'r': 0.3409090909090909}}]"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = \" \".join(map(lambda s: \" \".join(s), item.summarize(\"idf\")[:10]))\n",
    "rouge_score(truth, s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = \" \".join(map(lambda s: \" \".join(s), item.summarize(\"idf\")[:10]))\n",
    "rouge_score(truth, s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['we then describe two novel types of noise that are designed for structured feature spaces, and explain how they can be marginalized to efficiently compute 𝐖.in dropout noise, each feature is set to zero with probability p>.']\n"
     ]
    }
   ],
   "source": [
    "summary_cont = lxr.get_summary(sentences, threshold=None)\n",
    "print(summary_cont)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "sln-summarizer",
   "language": "python",
   "name": "sln-summarizer"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
